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PART  I 

SOFTWARE  OPERATOR-MACHINE  INTERFACE 
EVALUATOR'S  HANDBOOK 
EVALUATOR  GUIDELINES 


A.  GENERAL. 

1 .  Purpose. 

The  purpose  of  this  handbook  is  to  provide  to  the  evaluator 
information  needed  to  participate  in  the  Air  Force  Test  and  Evalu¬ 
ation  Center's  (AFTEC's)  software  operator-machine  interface  evalua¬ 
tion. 

This  evaluation  methodology  provides  a  standardized  approach 
to  determine  the  adequacy  of  that  part  of  software  design/implementa¬ 
tion  which  affects  interaction  between  a  computer-driven  system  and 
its  operator. 

2.  Handbook  Organization. 

This  handbook  is  divided  into  four  major  sections. 

a.  This  first  section  provides  the  evaluator  with: 

(1)  A  generalized  introduction  to  the  evaluation  concept. 

(2)  A  basic  understanding  of  the  evaluation  procedures. 

(3)  Background  information. 

(4)  Proccsdures  to  follow  to  recommend  changes  in  the 
evaluation  process  to  AFTEC. 

o.  The  second  section  of  the  handbook  contains  a  one-page 
explanation  of  each  "question"  used  in  the  evaluation.  The  pages  in 
this  section  are  numbered  OM-1  through  OM-95  to  correspond  to  the 
question  numbers.  In  addition,  pages  are  provided  to  separate 
subsections. 
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c.  The  third  section  of  the  handbook  contains  a  cross- 
reference  index  of  questions  vs  subject  matter  and  a  glossary  of 
terminology. 

d.  The  fourth  section  of  the  handbook  contains: 

(1)  The  Software  Operator-Machine  Interface  Question¬ 
naire  (short-form). 

(2)  A  sample  answer  sheet. 

(3)  An  AFTEC  Form  207,  Computer  Program  Observation 
Report. 

(4)  A  blank  question  data  sheet  to  be  used  in  recom¬ 
mending  changes  to  AFTEC. 

3.  Impetus. 

HQ  AFTEC,  in  an  effort  to  decrease  the  subjectivity  of  analyses 
of  the  software  portion  of  the  operator-machine  interface,  has  gen¬ 
erated  the  Software  Operator-Machine  Interface  Questionnaire 
(SOMIQ) .  The  concept  behind  the  SOMIQ  is  that  each  operator/ 
evaluator  is  directed  to  isolate  and  consider  each  of  a  number  of 
quality  factors  about  the  equipment  being  evaluated.  Through  this 
organized  approach,  the  operators  all  consider  the  same  subjects; 
thereby  yielding  a  uniform  analysis.  Furthermore,  the  operators  are 
guided  to  consider  subjects  which  they  might  overlook  if  asked  to 
generate  lists  of  problem  areas.  Additionally,  information  is  obtained 
as  to  which  aspects  of  a  system  contribute  positively  to  operational 
capabilities. 

4.  Background . 

In  the  past,  the  operator-machine  interface  for  a  piece  of 
computer-driven  equipment  has  been  evaluated  on  an  exception-only 
basis;  i.e.,  each  operator  would  address  only  those  areas  of  the 
interface  with  which  he  was  particularly  disturbed.  This  method  of 
analysis  naturally  resulted  in  highly  subjective,  non-specific  results. 
Operators  would  rate  the  interface  "good"  or  "bad"  according  to  the 
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number  and  difficulty  of  problems  each  encountered.  Experienced 
operators  often  have  less  problems  than  inexperienced  operators 
merely  because  they  have  learned  to  live  with  system  pecuiiarities. 

Highly  subjective  evaluations  are  undesirable  because  they  do 
not  yield  accurate  estimates  of  operational  capabilities  and  they  do 
not  yield  descriptions  of  specific  problems  that  need  to  be  fixed  to 
increase  operational  capabilities. 

The  SOMIQ  was  conceived  in  the  spring  of  1978  and  the  first 
version  was  published  in  the  winter  of  1979.  That  version  was 

advertised  widely  through  the  Air  Force  Systems  Command's  and  Air 

Force  Logistics  Command's  embedded  computer  resource  newsletters 
and  through  the  Automatic  Test  Equipment  (ATE)  newsletter  publish¬ 
ed  by  the  Navy.  Approximately  200  interested  parties  responded  to 
the  advertisements  and  about  half  that  number  provided  comments  to 
AFTEC.  In  addition,  the  SOMIQ  has  been  used  by  several  AFTEC 

test  teams  on  a  variety  of  systems  since  the  first  version  was 

published . 

B .  Evaluation  Methodology. 


The  methodology  for  evaluating  the  software  portions  of  the 
operator-machine  interface  is  based  on  the  use  of  a  closed  form 
questionnaire  with  optional  written  comments.  This  questionnaire  is 
designed  to  determine  the  extent  of  the  presence  of  certain  desirable 
attributes  in  a  given  system. 

The  desirable  attributes  addressed  by  the  questionnaire  are 
divided  into  six  groups  called  characteristics  (figure  1).  Further¬ 
more,  the  characteristics  of  assurability,  controllability,  and  workload 
reasonability  may  be  combined  to  form  a  test  factor  called  operability, 
while  descriptiveness,  consistency,  and  simplicity  may  be  combined 
to  form  a  test  factor  called  communicativeness.  A  complete  under¬ 
standing  of  the  definitions  of  the  characteristics  is  of  prime  import¬ 
ance  to  an  accurate  evaluation;  thus,  the  evaluator  should  study 
these  definitions  carefully. 


SOFTWARE  OPERATOR-MACHINE  INTERFACE 
QUESTIONNAIRE  CONCEPTUAL  MODEL 


(OFERAEILITY) 

—  ASSURABILITY 

-  CONTROLLABILITY 

-  WORKLOAD 

REASONABILITY 
v _ 


OPERATOR-MACHINE 

INTERFACE 


(COMMUNICATIVENESS) 

1-  DESCRIPTIVENESS 
CONSISTENCY 
SIMPLICITY 


CHARACTERISTICS 


Figure  1.  Software  Operator-Machine  Interface 
Questionnaire  Conceptual  Model 
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a.  Operability.  A  computerized  system  contains  the  quality  of 
operability  to  the  extent  that  the  operator  is  in  control  of  the 
operator-machine  interface.  Operability  is  the  sum  of  assurability , 
controllability,  and  workload  reasonability. 

(1)  Assurability.  A  computerized  system  contains  the  quality 
of  assurability  to  the  extent  that  it  aids  the  operator  in 
validating  data,  avoiding  errors,  and  correcting  errors 
once  made.  A  system  which  has  been  designed  to  aid  the 
operator  in  error  avoidance  may  or  may  not  have  good 
assurability.  A  system  should  also  be  designed  so  that 
errors  are  easy  to  correct,  and  above  all  so  that  errors  do 
not  have  catastrophic  effects. 

(2)  Controllability .  A  computerized  system  contains  the 
quality  of  controllability  to  the  extent  that  it  allows  the 
operator  to  direct  the  operations  of  the  machine.  The 
operator  must  be  able  to  direct  or  control  the  operation  of 
the  machine  in  order  to  utilize  it  effectively  and 
efficiently. 

(3)  Workload  reasonability.  A  computerized  system  contains 

the  quality  of  workload  reasonability  to  the  extent  that  the 
tasks  required  of  the  operator  are  within  the  operator's 
capability  and  require  the  operator  to  perform  a  useful, 
meaningful  role.  Optimum  design  of  a  system  which 

involves  an  operator  and  a  computerized  machine  takes 
advantage  of  the  best  capabilities  of  both:  the  machine  to 
perform  repetitive  tasks  rapidly  and  the  operator  to  make 
command  decisions  involving  unusual  situations. 

b.  Communicativeness.  A  computerized  system  contains  the 
quality  of  communicativeness  to  the  extent  that  the  transfer  of 
information  between  the  operator  and  the  machine  is  concise  and 
complete.  Communicativeness  is  the  sum  of  descriptiveness, 
consistency,  and  simplicity. 


(1)  Descriptiveness.  A  computerized  system  contains  the 
quality  of  descriptiveness  to  the  extent  that  the  operator 
has  available  adequate  explanations  of  every  function  the 
operator  is  required  to  perform,  and  every  function  the 
machine  performs.  The  operator  need  not  be  informed  in 
detail  of  every  task  the  machine  performs,  but  there 
definitely  are  certain  things  the  operator  must  know  to 
fulfill  the  mission.  The  questionnaire  relies  upon  the 
knowledge  of  the  operator  to  define  what  it  is  the  operator 
needs  to  know. 

(2)  Consistency.  A  computerized  system  contains  the  quality 
of  consistency  to  the  extent  that  the  behavior  of  the 
machine  and  documentation  corresponds  to  the  expectations 
of  the  operator.  There  should  be  a  near  one-to-one 
correspondence  between  what  the  machine  does,  what  the 
documentation  says  it  will  do,  and  what  the  operator  has 
been  trained  to  expect  the  machine  to  do.  Furthermore, 
the  documentation  normally  available  to  the  operator  should 
agree  with  that  which  the  operator  has  been  trained  to 
expect. 

(3)  Simplicity.  A  computerized  system  contains  the  quality  of 
simplicity  to  the  extent  that  information  presented  to  the 
operator  or  entered  by  the  operator  is  grouped  into  short, 
readily  understandable  structures.  Complicated  data 
structures,  data  entry  formats,  or  operator  manuals  all 
require  the  operator  to  spend  more  time  in  developing  an 
understanding  of  the  system,  and  may  have  a  tendency  to 
confuse  the  operator  as  well. 

2.  Environment. 

The  .-oftware  operator-machine  interface  evaluation  may  be 
conducted  in  one  or  more  of  several  areas  within  an  overall  weapons 
system  operational  test  and  evaluation.  The  evaluation  for  any  given 
system  may  involve  one  or  more  weapon  system  operators  as  well  as 


operators  for  one  or  more  levels  of  computer-driven  support  equip¬ 
ment.  Each  of  these  different  types  of  operator  functions  must  be 
evaluated  separately  from  the  other  since  the  design  of  the 
operator-machine  interface  is  different  for  each  type  of  operator. 

3.  When  to  Evaluate. 

It  is  left  to  the  software  test  manager  and  deputy  for  software 
evaluation  to  determine  timing  for  the  performance  of  this  evaluation. 
However,  it  is  appropriate  to  present  some  general  guidelines  and 
considerations: 

a.  One  of  the  assumptions  made  in  the  design  of  the  SOMIQ  is 
that  the  operators  performing  the  evaluation  are  familiar  with  the 
equipment  being  evaluated.  Therefore,  this  evaluation  should  not  be 
performed  before  the  evaluators  are  familiar  with  the  equipment. 

b.  It  is  quite  acceptable  to  perform  the  evaluation  more  than 
once  concerning  the  same  operator  station.  The  evaluation  might  be 
done  once  toward  the  middle  of  the  test  period  and  again  at  the  end 
of  the  test  period.  Reasons  are  to  see  what  effect  increased 
operator  experience  has  and  to  see  the  impact  of  contractor  updates. 

4.  Evaluators. 

One  of  the  major  assumptions  of  this  evaluation  methodology  is 
that  the  evaluators  have  a  working  knowledge  of  the  system  from  the 
perspective  of  the  specific  operator  station  being  evaluated.  The 
more  experience  an  operator  has,  the  more  accurate  the  judgements 
which  the  evaluator  must  make.  The  bottom  line  is  that  the  most 
experienced  operators  available  should  perform  the  evaluation. 

However,  if  the  number  of  highly  experienced  operators  is 
exceedingly  small  and  there  are  an  adequate  number  of  evaluators  of 
lesser  experience  available,  then  all  these  operators  should  perform 
the  evaluation.  HQ  AFTEC  can  run  the  analysis  of  results  by  both 
comparing  and  combining  the  results  from  both  groups  of  evaluators. 
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Occasionally,  the  situation  will  arise  in  which  there  are  no 

evaluators  with  any  actual  operator  experience.  This  might  occur 
when  no  hardware  has  been  produced  and  the  evaluation  is  being 
performed  on  a  proposal  or  design  document.  Such  a  situation  will, 
of  course,  produce  an  evaluation  extremely  dependent  on  the 
evaluator's  subjective  opinion,  experience,  and  interpretation  of  the 
documentation  available. 

5.  Procedures. 

The  software  operator-machine  interface  evaluation  consists  of 
four  phases:  planning,  calibration,  assessment,  and  analysis. 

a.  During  the  planning  phase,  the  deputy  for  software 

evaluation  and  the  software  test  manager  identify  the  operator- 
machine  interface  is  to  be  evaluated,  when  and  where  the  evaluation 
is  to  take  place,  and  who  the  evaluators  are  to  be.  In  addition, 
they  are  responsible  for  assuring  that  required  resources  are  avail¬ 
able  to  the  evaluators  at  evaluation  time. 

b.  During  the  calibration  phase,  the  evaluators  receive  a 

detailed  procedures  briefing  on  use  of  the  questionnaire.  During 

this  briefing,  each  question  in  the  questionnaire  is  discussed  in 

detail.  It  is  extremely  important  that  each  evaluator  leave  the 
briefing  with  a  thorough  understanding  of  each  and  every  question. 

c.  The  assessment  phase  is  the  period  of  time  during  which 
the  evaluators  complete  the  questionnaire.  This  phase  will  require 
one  day  or  less  for  each  evaluator  per  questionnaire  completed. 

d.  During  the  analysis  phase,  the  deputy  for  software 
evaluation  will  collect  the  completed  questionnaires,  perform  a  quality 
check,  and  forward  the  answer  sheets  to  HQ  AFTEC  for  data 
reduction  and  a  preliminary  analysis  of  results.  The  results  will  be 
returned  to  the  deputy  for  software  evaluation  for  final  analysis  and 
inclusion  in  operational  test  and  evaluation  reports. 


6.  Evaluation  Concepts. 


In  performing  the  software  operator-machine  interface  evaluation 
there  are  several  important  concepts  which  the  evaluator  must  keep 
in  mind. 


a.  Questions.  The  "questions"  in  the  questionnaire  are  not 
questions  at  all.  Each  question  is  a  positive  statement  about  a 
desirable  design  feature  as  it  would  be  implemented  in  an  ideal 
system.  The  question  is  answered  by  the  evaluator  indicating  how 
well  the  system  under  evaluation  lives  up  to  the  ideal  described  in 
the  question. 

b.  Response  Scale.  The  following  response  scale  will  be  used 


to  answer  each 

question : 

1) 

A. 

Completely  agree  (absolutely  no  doubt). 

2) 

B. 

Strongly  agree. 

3) 

C. 

Generally  agree. 

4) 

D. 

Generally  disagree. 

5) 

E. 

Strongly  disagree. 

6) 

F. 

Completely  disagree  (absolutely  no  doubt). 

One  of  these 

responses  must  be  given  for  each  question.  In 

addition,  one 

or 

more  of  the  following  standardized  comment 

responses  can 

be  selected: 

1) 

1. 

1  had  difficulty  answering  this  question. 

2) 

J. 

A  written  comment  has  been  submitted. 

The  responses  G  and  H  do  not  currently  have  any  meaning. 

The  responses 

A 

to  F  indicate  the  extent  to  which  the  evaluator 

agrees/disagrees  with  the  question  statement.  Depending  on  the 
application  area  and  the  type  of  question,  these  responses  can  be 
interpreted  differently.  In  general,  however,  the  response  scale  can 
be  interpreted  as  follows: 

1)  A.  Completely  agree.  There  must  be  absolutely  no 
doubt  when  using  this  response  that  the  product 
being  evaluated  cannot  be  any  better  with  respect  to 
the  characteristic  addressed. 
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2)  B.  Strongly  agree.  This  response  indicates  that 

the  product  being  evaluated  is  very  good  and  very 
helpful  to  the  operator. 

3)  C.  Generally  agree.  This  response  indicates  that 

the  product  being  evaluated  is  acceptable  and  helpful 
to  the  operator. 

4)  D.  Generally  disagree.  This  response  indicates 

that,  although  the  product  being  evaluated  is  accept¬ 
able,  some  improvements  are  required  to  make  it 
helpful  to  the  operator. 

5)  E.  Strongly  disagree.  This  response  indicates  that 

the  product  being  evaluated  is  unacceptable  and  major 
improvements  are  required  before  it  would  be  helpful 
to  the  operator. 

6)  F.  Completely  disagree.  There  must  be  absolutely 

no  doubt  when  using  this  response  that  the  product 
being  evaluated  is  unacceptable  and  must  be  com¬ 
pletely  redesigned  or  rewritten  to  be  acceptable  with 
respect  to  the  characteristic  addressed. 

c.  Not  Applicable.  Occasionally  the  evaluator  will  encounter  a 
question  that  does  not  seem  to  apply  to  the  system  being  evaluated. 
In  this  circumstance,  the  evaluator  will  have  to  work  harder  at 
formulating  an  answer.  When  a  question  seems  to  be  not  applicable, 
it  is  usually  because  some  feature  or  function  of  the  idealized  system 
is  entirely  absent  in  the  system  under  consideration.  In  this  case 
the  evaluator  must  formulate  an  answer  based  upon  any  system 
deficiency  caused  by  the  lack  of  the  desired  feature.  If  operation  of 
the  system  would  be  enhanced  by  the  presence  of  the  missing 
feature,  then  the  question  should  be  answered  in  the  D,  E,  or  F 
range.  If,  on  the  other  hand,  presence  of  the  missing  feature 
would  not  help  the  operator,  the  question  should  be  answered  in  the 
A,  B,  or  C  range.  The  reason  that  the  answer  in  such  "not  appli¬ 
cable"  cases  is  not  always  A  or  F  is  that  the  missing  feature  may  be 
a  nice-to-have  item.  It  is  up  to  the  evaluator  to  judge  how 
seriously  lack  of  certain  features  effect  the  operator's  ability  to 
complete  the  mission. 
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d.  Keep  the  Characteristic  in  Mind.  When  answering  a 
question  in  a  given  characteristic  group,  answer  the  question  only 
as  it  applies  to  that  characteristic  group  and  the  definition  of  that 
characteristic.  For  instance,  question  9  (the  data  entry  display  has 
a  cursor  or  pointer)  should  only  be  answered  as  to  how  effective  the 
display  is  in  telling  the  operator  where  his  next  character  will  be 
typed  in  a  line  of  text,  thereby  avoiding  the  error  of  no  blanks  or 
too  many  blanks.  True,  this  contains  a  certain  amount  of  descrip¬ 
tiveness  (as  do  almost  all  questions),  but  the  emphasis  here  is  on 
assurability  (error  avoidance  in  this  case). 

e.  Take  Your  Time.  Heavy  emphasis  must  be  placed  upon  the 
best,  most  carefully  considered  answers  you  can  give.  A  few  extra 
minutes  of  careful  thought  right  now  may  easily  save  the  Air  Force 
hundreds  of  hours  of  needless  hassle  later  on.  System  deficiencies 
revealed  now  can  be  corrected  before  the  system  is  in  field  use. 

f.  Use  the  Comment  Sheets.  Everything  said  in  the  previous 
paragraph  applies  equally  well  here.  This  questionnaire,  being  a 
human  product,  cannot  possibly  be  expected  to  reveal  every  shortfall 
of  every  system.  Your  knowledge  as  expressed  in  the  comment 
sheets  (AFTEC  Form  207)  is  the  only  way  that  this  analysis  can 
reveal  deficiencies  not  specifically  or  exactly  addressed  by  questions. 

7.  How  to  Recommend  Changes. 

Since  it  is  the  considered  opinion  of  AFTEC  that  this  manual  is 
not  a  "perfect"  test  tool  (nor  will  it  ever  be),  AFTEC  must  be 
prepared  to  change  the  document  as  time  passes.  One  of  the  best 
sources  of  additional  information  to  be  included  in  this  manual  is  the 
very  people  who  use  it.  Therefore,  a  blank  question  data  sheet  has 
been  provided  in  section  IV  to  be  used  as  a  medium  with  which  the 
evaluator  may  forward  recommended  changes  to  AFTEC.  The 
question  data  sheet  may  be  used  to  address  exact  questions  (fill  in 
the  question  number),  or  to  suggest  new  questions.  Please  send  the 
question  data  sheet  along  with  any  additional  information  to: 
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HQ  AFTEC/TEBC 
Kirtland  AFB  NM  87117 

Please  submit  a  Computer  Program  Observation  Report  with  the 
question  data  sheet  to  identify  yourself  and  the  circumstances  which 
lead  to  your  recommendation.  AFTEC  personnel  will  contact  you  and 
discuss  the  recommendation  with  you. 
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PART  II 


SOFTWARE  OPERATOR-MACHINE  INTERFACE 
EVALUATOR'S  HANDBOOK 

QUESTION  RESPONSE  GUIDELINES 


The  following  sections  contain  information  which  should  help 
clarify  the  intent  of  each  question  to  the  evaluator.  There  is  one 
section  per  characteristic.  Each  page  within  each  section  corre¬ 
sponds  to  a  question  from  the  Software  Operator-Machine  Interface 
Questionnaire.  Many  questions  have  special  response  instructions 
which  should  be  reviewed. 


OM-O 


A.  ASSURABILITY.  A  computerized  system  contains  the  quality  of 
assurability  to  the  extent  that  it  aids  the  operator  in  validating  data, 
avoiding  errors,  and  correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error 
avoidance  may  or  may  not  have  good  assurability.  We  all  make 
mistakes  anyhow.  A  system  should  also  be  designed  so  that  errors 
are  easy  to  correct,  and  above  all  so  that  errors  do  not  have  cata¬ 
strophic  effects. 


QUESTION  DATA  SHEET 


Question  Number  1 


QUESTION :  Operator  input  errors  do  not  cause  system  failures. 
CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  Commands  and  data  entered  by  the  operator  should 
not  cause  hardware  or  software  errors  inside  the  machine. 

EXAMPLES :  A  bad  command  by  the  operator  could  be  a  request  to 
the  computer  to  connect  a  12-VDC  power  supply  to  the  same  bus 
that  a  5-VDC  power  supply  is  connected.  Another  bad  command  by 
the  operator  could  be  entering  an  "8"  when  the  computer  could  be 
requesting  a  number  between  "1"  and  "7."  Other  bad  commands 
could  be  instructions  to  start  a  device  which  is  already  running  or 
stop  one  which  is  already  stopped.  The  reason  that  these  are  not 
"don't  care"  situations  is  that  the  operator  may  have  intended  to 
start  or  stop  a  different  device. 

GLOSSARY :  Operator  input:  Data  or  commands  entered  as  type¬ 
written  text,  pushbutton  sequences,  control  stick  movements,  card 
decks,  tapes,  etc. 


QUESTION  DATA  SHEET 


Question  Number  2 


QUESTION :  Operator  input  errors  are  detected. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  The  machine  must  be  designed  to  detect  operator 
input  errors  or  it  will  try  to  function  using  bad  data.  Anything  the 
machine  does  based  on  the  bad  data  may  be  wrong,  as  well  as  pos¬ 
sibly  dangerous  to  both  equipment  and  personnel. 

EXAMPLES: 


GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  3 


QUESTION :  The  causes  of  input  errors  are  displayed  to  the  opera¬ 
tor. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  When  an  operator  has  made  an  error,  the  operator 
needs  to  know  what  the  error  was  in  order  to  correct  it. 

EXAMPLES:  It  is  insufficient  for  a  machine  to  display  some  message 
such  as  "input  error".  Such  a  message  does  not  tell  the  operator 
what  was  wrong,  only  that  something  was  wrong. 

GLOSSARY: 


* 

« 
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QUESTION  DATA  SHEET 


Question  Number  4 


QUESTION :  The  action  required  to  correct  an  operator  input  error 
is  displayed  to  the  operator. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  Having  the  machine  tell  the  operator  what  to  do  to 
correct  an  error  is  much  quicker  than  having  the  operator  refer  to 
manuals,  etc.,  to  look  for  the  proper  procedure. 


EXAMPLES:  The  machine: 

The  operator: 
The  machine: 
The  operator: 
The  machine: 


ENTER  DATE 
Jun  16,  1980 
NO,  FORMAT: 
80,  06,  16 
OK 


VV,  MM,  DD 


GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  5 


QUESTION :  Input  errors  are  easily  corrected. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  The  actions  necessary  to  correct  errors  can  vary 

all  the  way  from  completely  reinitializing  the  system  to  simply  correct¬ 
ing  the  erroneous  portion  of  a  command.  It  is  desirable  from  an 
assurability  point  of  view  that  correction  of  operator  errors  be  as 
simple,  easy,  and  straight-forward  as  possible. 

EXAMPLES:  If  the  operator  enters  a  long  command,  such  as  "Load 
Program  TESTLRU17  frm  disk  22,"  the  machine  should  allow  the 
operator  to  correct  a  misspelling  easily  rather  than  retype  the  entire 
command  sequence. 

The  machine  would  redisplay  the  incorrect  command  (above)  and 
allow  the  operator  to  insert  the  "o"  in  the  word  "frm"  to  create  the 
word  "from,"  and  then  accept  the  corrected  command. 

GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  6 


QUESTION :  Input  errors  are  quickly  corrected. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  For  an  error  to  be  termed  "quickly  correctable," 
the  system  must  resume  normal  operation  immediately  after  the  error 
is  corrected. 

EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  7 

QUESTION :  The  operator  can  verify  input  before  execution/entry. 
CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  If  the  operator  can  see  the  instruction  as  it  is 
entered,  then  mistyped  characters,  etc.,  can  be  corrected  on-the- 
spot  rather  than  after  the  machine  indicates  incorrect  input. 

EXAMPLES: 

GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  8 


QUESTION :  Mission  peculiar  data  entered  by  the  operator  is  checked 
for  validity. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS :  When  the  operator  enters  mission  data  (altitude, 
voltage,  range)  the  data  should  be  checked  by  the  machine  as  much 
as  feasible  to  ensure  that  the  mission  is  not  completed  with  wrong 
data,  resulting  in  reaccomplishment. 

EXAMPLES:  An  aircraft  simulator  operator  enters  a  simulated  terrain 
altitude  of  60,000  ft  when  it  should  have  been  6,000  ft,  thereby, 
causing  a  "crash"  and  subsequent  restart  of  the  simulation.  The  data 
should  have  been  checked  against  a  maximum  of,  say,  26,000  ft. 


ADDITIONAL  EXAMPLES: 
ENTRY 

100  degrees  North  latitude 
32  Jan  80 
Mach  12 


(SHOULD  HAVE  BEEN) 
(10.0  N  Lat) 

(31  Jan  80) 

(Mach  1.2) 


GLOSSARY :  Mission  peculiar:  any  data  which  is  entered  by  the 
operator  as  a  part  of  normal  operations  but  which  may  change 
mission-to-mission  or  task-to-task. 
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QUESTION  DATA  SHEET 


Question  Number  9 


QUESTION :  The  data  entry  display  has  a  cursor  or  pointer. 
CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  The  operator  must  know  where  the  next  character 
entered  will  be  placed  to  avoid  errors  caused  by  too  many  spaces  or 
'double'  errors  caused  by  correcting  the  wrong  character. 

EXAMPLES:  The  operator  enters  "REWIID  TAPE  22,"  then  notices 
the  error.  The  system  allows  him  to  backspace  and  change  "REWIID" 
to  "REWIND."  A  cursor  allows  him  to  backspace  quickly  and  accu¬ 
rately.  Lack  of  a  cursor  causes  him  to  count  backspaces  and  per¬ 
haps  change  "REWIID"  to  "REWNID." 

The  communications  system  operator  in  an  E-4B  is  entering  a 
command  when  the  aircraft  suddenly  hits  turbulance  and  the  opera¬ 
tor's  finger  hits  the  space  bar  several  times.  With  improper  spacing, 
the  command  will  be  formatted  incorrectly  and  the  machine  will  reject 
the  command,  forcing  the  operator  to  re-enter.  A  cursor  would 
allow  the  operator  to  see  how  many  extra  spaces  were  entered,  enter 
the  appropriate  number  of  backspaces,  and  resume  entry  where  it 
was  interrupted. 

GLOSSARY :  Cursor  or  pointer:  a  method  of  indicating  on  the 
display  where  the  next  character  to  be  entered  will  appear  on  the 
display. 


QUESTION  DATA  SHEET 


Question  Number  10 


QUESTION :  The  operator  is  able  to  correct  mistyped  characters 
with  a  backspace  key. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  It  is  much  faster  and  less  error  prone  to  be  able 
to  backspace  and  change  an  incorrect  character  than  it  is  to  be 
forced  to  retype  an  entire  entry. 

EXAMPLES: 


GLOSSARY :  Backspace:  rubout,  back  arrow,  etc. 


QUESTION  DATA  SHEET 


Question  Number  11 


QUESTION :  The  system  does  not  require  the  operator  to  copy 
information  by  hand. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  Hand  copying  information  from  a  computer  display 
to  some  form  of  record  is  a  source  of  errors  and  extra  work  which 
could  be  handled  by  the  machine. 

EXAMPLES :  An  operator  is  displaying  satellite  telemetry  points  on  a 
screen  and  desires  some  form  of  hardcopy.  If  the  operator  is  forced 
to  copy  the  information  by  hand  as  the  data  is  changing,  the  data  is 
useless  since  the  numbers  copied  down  will  not  all  be  taken  at  the 
same  time.  The  operator  should  have  some  kind  of  "picture  taking" 
capability  which  would  automatically  produce  printed  forms  of  the 
data. 

GLOSSARY: 


4 


QUESTION  DATA  SHEET 


Question  Number  12 


QUESTION :  Task  aborts  and  interrupts  do  not  have  detrimental  side 
effects . 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  Turning  the  machine  off  for  an  emergency  power 

failure  or  stopping  a  test,  etc.,  should  not  result  in  mechanical 
damage  or  loss  of  data  files  which  would  be  difficult  to  replace. 

EXAMPLES:  The  machine  is  performing  a  test  from  a  tape  drive. 

Power  failure  causes  the  tape  drive  to  tear  the  tape  in  half.  The 
tape  can  only  be  replaced  by  a  new  tape  from  the  depot. 

GLOSSARY: 

Task  aborts:  emergency  shutdown;  (simulator)  mission  restart; 
and  a  variety  of  other  reasons  for  wanting  to  simply  quit  at  any  time 
other  than  when  the  machine  would  normally  expect. 

Task  interrupt:  pause,  freeze,  or  even  "hold  it-it's  time  for 
my  coffee  break." 


s. 


QUESTION  DATA  SHEET 

Question  Number  13 


QUESTION :  Selecting  a  device  off-line  does  not  have  detrimental 

side  effects. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  When  an  operator  manually  takes  a  device  off-line, 
he  generally  has  a  good  reason.  Such  action  should  not  interrupt 
system  operation  except  as  it  pertains  to  that  device. 

EXAMPLS:  An  operator  is  waiting  for  a  long  test  to  be  completed. 

The  printer  is  not  in  use,  but  the  operator  knows  it  is  short  of 
paper.  Taking  the  printer  off-line  to  replace  the  paper  should  not 
cause  the  test  to  stop. 

A  weapon  system  operator  shuts  off  one  display  scope  which  is 
obviously  malfunctioning  and  is  distracting  him.  Shutting  this  one 
scope  off  should  not  cause  any  other  changes  to  the  system,  and 
should  certainly  not  cause  the  whole  electronic  warfare  system  to 
quit. 

GLOSSARY :  Off-line:  out  of  operation,  power  off,  or  de-selection. 


QUESTION  DATA  SHEET 
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Question  Number  14 


QUESTION :  The  system  automatically  ceases  execution  if  internal 

errors  are  detected. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  When  the  computer  detects  an  error,  it  should 

immediately  halt  all  executions  to  avoid  damage  to  data  files  and  to 
avoid  giving  wrong  answers. 

EXAMPLES:  A  data  word  stored  in  the  computer  internal  storage 

represents  altitude  above  ground  level.  When  recalled,  the  word 
had  a  read  or  parity  error.  If  the  word  were  interpreted  and  sent 
to  the  pilot  it  might  tell  him  2000  ft,  whereas,  the  correct  altitude 
might  be  50  ft. 

GLOSSARY:  Parity:  a  method  of  checking  the  validity  of  data,  A 
parity  error  indicates  invalid  data. 
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QUESTION  DATA  SHEET 

Question  Number  15 


QUESTION :  The  operator  is  alerted  to  faults  within  the  system. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  This  question  is  aimed  at  eliminating  operator  work 
having  to  be  redone  once  the  fault  is  discovered.  The  software 
should  be  designed  to  detect  as  many  failures  as  possible  and  to 
report  them  to  the  operator  as  soon  as  possible.  Any  unscheduled 
change  in  the  status  of  any  part  of  the  system  should  be  reported  to 
the  operator. 

EXAMPLES: 

1.  The  operator  is  entering  a  long  sequence  of  instructions 
and  does  not  realize  the  machine  is  in  an  unusable  status.  When  he 
finally  finds  out  the  machine  is  "broken,"  he  must  turn  it  off,  repair 
it,  then  reenter  the  long  sequence  of  commands. 

2.  It  is  very  nice  for  a  pilot  to  know  when  the  terrain  avoid¬ 
ance  radar  ceases  to  function  or,  even  worse,  starts  to  give  errone¬ 
ous  data. 

GLOSSARY: 

Diagnostic:  fault  detection  and  reporting  hardware. 

Faults:  burned  out  power  supply,  out-of-calibration  stimulus 

voltage,  etc. 
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QUESTION  DATA  SHEET 


Question  Number  16 


QUESTION :  The  causes  of  system  halts  are  displayed  to  the  opera¬ 
tor. 

CHARACTERISTIC:  Assurability . 

EXPLANATIONS:  This  is  important  information  for  anyone  who  must 
fix  or  restart  the  system. 

EXAMPLES:  An  aircraft  pilot  needs  to  know  whether  the  navigation 
computer  shut  itself  down  due  to  equipment  bay  overheat,  faulty 
programming,  or  just  random  error.  Restarting  the  computer  in  a 
hot  compartment  could  damage  the  equipment.  Restarting  the  com¬ 
puter  to  execute  a  scrambled  program  would  be  useless  or  possibly 
dangerous.  Restarting  the  computer  after  a  random  error  would 
probably  be  OK. 

GLOSSARY: 
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ASSURABILITY.  A  computerized  system  contains  the  quality  of 
assurability  to  the  extent  that  it  aids  the  operator  in  validating 
data,  avoiding  errors,  and  correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error 
avoidance  may  or  may  not  have  good  assurability.  We  all  make 
mistakes  anyhow.  A  system  should  also  be  designed  so  that  errors 
are  easy  to  correct,  and  above  all  so  that  errors  do  not  have  cata¬ 
strophic  effects. 


8.  CONTROL  LA  81  LITV.  A  computerized  system  contains  the  quality 
of  controllability  to  the  extent  that  it  allows  the  operator  to  direct 
the  operations  of  the  machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of 
the  machine  in  order  to  utilize  it  effectively  and  efficiently. 


I 

i 

I 
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QUESTION  DATA  SHEET 


Question  Number  17 


QUESTION :  The  operator  can  interrupt  and  resume  automatic  pro¬ 
cesses  . 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  It  is  often  desirable  for  the  operator  to  halt 
execution  of  a  process,  change  some  parameters,  and  then  proceed 
with  the  process. 

EXAMPLES:  A  test  equipment  operator  notices  that  while  both  a 
voltage  and  resistance  measurement  are  within  tolerance,  the  voltage 
is  borderline  high  and  the  resistance  is  borderline  low.  The  opera¬ 
tor  interrupts  automatic  test,  calculates  current,  and  sees  that  it  is 
actually  still  within  tolerance.  Automatic  test  is  then  resumed. 

GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  18 


QUESTION :  The  operator  has  task  abort  capabilities  available. 
CHARACTERISTIC  Controllability. 

EXPLANATIONS:  It  is  often  necessary  to  stop  a  task  before  it  is 
completed  because  of  obvious  errors  or  higher-priority  workload. 

EXAMPLES:  A  mission  preparation  operator  is  printing  out  a  table 
of  latitudes  and  longtitudes  when  he  notices  that  the  first  entry  is 
in  error.  He  must  wait  for  the  entire  table  to  print  out  before  he 
can  correct  the  error  and  reprint  because  there  is  no  abort  capabil¬ 
ity.  If  he  had  an  abort  capability  he  could  abort  the  print,  fix  the 
error,  and  then  restart  the  print  immediately. 

GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  19 


QUESTION :  The  operator  may  initiate  selected  self-test  systems. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Automated  troubleshooting  procedures  are  a  great 
help  to  operators,  but  not  all  of  them  may  be  used  continuously. 
Therefore,  some  of  them  must  be  initiated  by  the  operator  at  times 
when  they  would  not  interfere  with  normal  system  operation. 

EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  20 


QUESTION :  The  operator  may  send  output  data  to  various  devices. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS :  Giving  the  operator  control  over  where  to  send 
output  data  allows  the  operator  greater  flexibility  in  completing  the 
mission. 

EXAMPLES: 

1.  A  simulator  operator  has  a  malfunctioning  display  screen.  If 
the  operator  can  route  the  output  that  should  go  to  that  screen  to 
some  other  screen,  then  the  simulator  can  be  used.  If  the  output 
cannot  be  rerouted,  then  perhaps  the  simulator  cannot  be  used  at  all 
and  the  entire  training  schedule  is  thrown  off. 

2.  An  automatic  test  station  has  a  printer  which  is  out  of 
paper.  If  the  operator  can  divert  the  printed  output  to  the  display 
screen,  the  necessary  repair  information  can  at  least  be  copied  down 
by  hand  and  the  item  can  be  fixed.  Otherwise,  maintenance  activi¬ 
ties  cease  until  paper  for  the  printer  is  procured. 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  21 


QUESTION :  The  operator  can  ask  for  and  receive  the  current 

status  of  operation. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS :  Especially  on  long  tasks,  it  is  necessary  for  the 
operator  to  know  that  things  are  proceeding  as  they  should.  Other¬ 
wise,  the  operator  might  wait  until  the  end  of  a  2-hour  task  only  to 
find  out  that  it  has  to  be  redone. 

EXAMPLES: 

1.  A  test  equipment  operator  is  performing  a  6-hour  alignment 

test  on  an  inertial  platform.  After  15  minutes,  he  queries  the  test 

equipment  system  status  and  sees  that  the  machine  has  been  doing 
nothing  but  trying  to  connect  a  power  supply  which  is  not  turned 
on.  Since  he  could  query  the  system  status  he  discovered  the 
situation  almost  immediately  rather  than  6  hours  later. 

2.  For  simulators,  the  entire  purpose  of  the  instructor  console 

is  usually  to  give  the  operator  the  current  status  of  operations.  It 

is  unlikely  that  an  instructor's  console  would  score  poorly  on  this 
question . 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  22 


QUESTION :  The  operator  can  select  the  type  of  information  shown 

on  the  display. 

CHARACTISTIC:  Controllability. 

EXPLANATIONS:  The  operator  must  be  able  to  control  what  type  of 
information  is  displayed  so  that  desired  information  may  be  obtained 
in  as  easily  understandable  a  method  as  possible. 

EXAMPLES: 

1.  A  student  pilot  is  flying  his  aircraft  on  a  strafing  run. 
There  are  so  many  arrows,  lines,  and  indicators  on  his  heads-up 
display  that  he  is  becoming  confused.  He  tabs  the  "declutter" 
button  and  notices  immediately  that  a  "low  airspeed"  warning  had 
been  hidden  amongst  all  the  garbage.  He  corrects  the  situation  and 
completes  the  mission. 

2.  A  satellite  analyst  has  displays  available  for  each  satellite 
subsystem.  The  power  subsystem  has  been  showing  unusual 
fluctuations.  Activity  in  the  data  relay  subsystem  is  the  suspected 
cause.  The  analyst  builds  a  new  display  with  some  telemetry  from 
both  the  power  subsystem  and  the  data  relay  subsystem  and  verifies 
that  the  power  fluctuations  are  the  result  of  a  faulty  transmitter 
turning  on  and  off. 

GLOSSARY: 

Displays 


QUESTION  DATA  SHEET 


Question  Number  23 


QUESTION:  The  operator  may  command  various  displays  of  system 


CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  operator  must  often  troubleshoot  problems  to 
correct  bad  situations.  To  do  this,  detailed  information  on  the 
status  of  various  systems  and  subsystems  must  be  available. 

EXAMPLES: 

1.  A  data  processing  computer  operator  notices  that  his  system 
has  seemingly  stopped.  A  system  status  check  shows  everything 
normal.  A  check  of  peripherals  shows  nothing  wrong.  A  check  of 
job  control  shows  that  one  small  computer  program  is  using  all  the 
system  time.  A  check  of  the  program  status  log  shows  no  input  and 
no  output.  The  operator  sees  that  another  novice  programmer  has 
written  another  infinite  loop.  The  operator  aborts  the  program,  and 
the  normal  processing  of  paychecks  resumes. 

2.  At  a  change  of  shift  on  a  weapon  control  system,  the  new 
operator  would  normally  want  to  display  the  current  status  of  all 
sensors,  communications  lines,  system  armament  resources,  etc. 

GLOSSARY: 


OM-23 


QUESTION  DATA  SHEET 


Question  Number  24 


QUESTION :  The  operator  may  control  the  amount  of  explanatory 
text,  both  input  and  output. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Explanatory  text  is  often  helpful  to  a  beginning 
operator,  but  it  is  just  extra  trash  on  the  display  to  an  experienced 
operator. 

EXAMPLES:  One  system  has  two  commands  which  control  this  feature 
The  command  “VERBOSE"  tells  the  machine  to  display  information 
with  plenty  of  explanatory  text.  The  command  "CONCISE"  tells  the 
machine  to  display  the  bare  essentials. 

GLOSSARY: 


OM-24 


ques i  ion  data  sheet 


Question  Number  25 


QUESTION :  The  operator  may  edit  the  data  base  prior  to  use  by 

the  system. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Many  systems  have  a  "scratchpad"  data  base  which 
must  be  edited  by  the  operator  to  create  a  data  base  which  applies 
to  the  current  task. 

EXAMPLES: 

Mission  Data  Preparation:  Prepare  a  bombing  run  with  standard 
ingress  and  egress  route,  changing  the  target. 

Simulators:  Prepare  a  mission  scenario  involving  an  air  inter¬ 

cept  using  non-standard  differential  altitude. 

Automatic  Test  Equipment:  Change  some  test  parameters  to 
reflect  that  the  LRU  undergoing  test  is  a  specially  modified  box. 

GLOSSARY:  Data  base:  a  set  of  parameters  used  by  a  computer 

program  as  opposed  to  the  computer  program  itself. 


OM-25 


QUESTION  DATA  SHEET 


Question  Number  26 


QUESTION :  The  operator  may  create  and  execute  strings  of  com¬ 
mands  as  a  single  command. 

CHARACTERISTIC:  fcontrollability . 

EXPLANATIONS:  The  operator  must  often  perform  repetitive  tasks. 
It  is  desirable  that  the  operator  be  able  to  reduce  the  workload  of 
repetitious  tasks  as  much  as  possible. 

EXAMPLES: 

1.  The  operator  must  perform  a  series  of  10  instructions  a 
great  number  of  times.  If  he  can  create  a  file  containing  the  10 
instructions  and  then  use  one  instruction  to  execute  the  entry  file, 
his  workload  is  reduced  by  a  factor  of  10. 

2.  So  called  "function  keys"  are  an  example  of  built-in  work 
reduction  techniques.  If  the  operator  can  program  the  function  keys 
to  perform  a  specific  task,  then  the  operator  has  the  desired  control. 

GLOSSARY: 


OM-26 


QUESTION  DATA  SHEET 

Question  Number  27 


QUESTION:  The  operator  can  command  visibility  (stepped  execution) 
of  automatic  processes. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  capability  for  stepped  execution  of  automatic 
processes  gives  the  operator  detailed  information  about  what  is 
occurring  during  a  normally  automatic  process. 

EXAMPLES: 

1.  An  automatic  test  equipment  operator  is  having  trouble  with 
a  supposedly  "good"  black  box  failing  an  automatic  test.  He  reruns 
the  test  in  stepped-execution  mode  which  gives  him  time  to  read 
every  action  the  machine  performs.  He  sees  that  one  part  of  the 
test  measures  a  voltage  at  5.2  volts.  The  test  program  fails  the 
box,  stating  the  voltage  should  be  5.0  ±  .1  volts.  Documentation 
states  the  voltage  should  be  5.0  ±  .25  volts.  The  box  is  good,  the 
test  program  is  wrong. 

2.  A  similar  visibility  function  within  an  aircraft  cockpit  would 
be  the  capability  provided  by  the  gun  camera.  Within  a  simulator  it 
would  be  videotape  replay. 

GLOSSARY: 


OM-27 


QUESTION  DATA  SHEET 


Question  Number  28 


QU ESTION :  The  operator  may  command  different  modes  of  oper¬ 
ation  . 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  capability  to  command  different  modes  of 
operation  gives  the  operator  the  ability  to  use  the  machine  in  a 
manner  consistent  with  his  need  and  experience  level. 

EXAMPLES: 

Aircraft:  The  ability  to  drop  bombs  in  single,  pairs,  or  ripple 
mode  as  the  pilot  deems  appropriate. 

Simulator:  The  capability  to  operate  two  side-by-side  simulators 
as  opposing  forces,  one  complete  crew,  or  as  entirely  separate 
simulators. 

Automatic  Test  Equipment:  Automatic  versus  stepped  execution 
testing. 

GLOSSARY: 


OM-28 


QUESTION  DATA  SHEET 


Question  Number  29 


QUESTION:  The  operator  may  control  the  type  and  quantity  of 

output. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Control  of  type  and  quantity  of  output  gives  the 

operator  control  over  what  the  operator  must  read.  Too  much  out¬ 
put,  or  output  the  operator  doesn't  even  want  to  see,  just  slows 
things  down. 

EXAMPLES: 

1.  Under  normal  operations  the  analyst  may  desire  to  watch 
the  CRT  and  shut  off  the  printer. 

2.  In  order  to  concentrate  on  the  important  parameters  during 
a  satellite  maneuver,  the  analyst  can  selectively  prevent  some  of  the 
other  parameters  from  printing  on  the  CRT  during  the  maneuver. 

GLOSSARY: 


OM-29 


QUESTION  DATA  SHEET 


Question  Number  30 


QUESTION :  Bypass  procedures  are  available  so  that  in  cases  of 

partial  system  failure  the  more  important  system  functions  can  still 
be  performed. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  It  is  often  desirable  to  get  the  job  done  even 

though  the  usual  tools  are  not  available.  This  question  refers  to 
degraded  modes  of  operation. 

EXAMPLES: 

1.  An  automatic  test  equipment  operator  sees  that  his  test  has 
halted  because  the  "test  station  fault"  light  is  on.  Researching  the 
problem,  he  sees  that  the  test  program  was  requesting  the  B5  power 
supply  be  connected  to  the  unit  under  test.  The  B5  power  supply 
was  sent  to  PMEL  the  previous  week.  He  causes  the  test  program  to 
substitute  the  B18  power  supply  which  can  do  the  same  job  as  the 
B5.  The  test  proceeds. 

2.  A  simulator  has  a  electronic  warfare  office  (EWO)  station. 
The  EWO  station  is  down  for  repair.  It  is  highly  desirable  to  be 
able  to  continue  training  of  other  crew  members. 

3.  Aircraft:  It  sure  is  nice  to  be  able  to  fly  home  even  if  the 
navigational  computer  quits. 

GLOSSARY: 


OM-30 


CONTROLLABI LITY .  A  computerized  system  contains  the  quality  of 
controllability  to  the  extent  that  it  allows  the  operator  to  direct  the 
operations  of  the  machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of 
the  machine  in  order  to  utilize  it  effectively.  The  operator  must  and 
effeciently. 


C.  WORKLOAD  REASONABILITY.  A  computerized  system  contains 
the  quality  of  workload  reasonability  to  the  extent  that  the  tasks 
required  of  the  operator  are  within  the  operator's  capabilities  and 
the  extent  to  which  the  operator  performs  a  useful,  meaningful  role. 

An  optimum  design  of  a  system  which  involves  an  operator  and 
a  computerized  machine  takes  advantage  of  the  best  capabilities  of 
both:  the  machine  to  perform  repetitive  tasks  rapidly  and  the  oper¬ 
ator  to  make  command  decisions  involving  unusual  situations. 


QUESTION  DATA  SHEET 


Question  Number  31 


QUESTION:  It  is  easy  to  enter  mission  (task)  peculiar  data. 
CHARACTERISTIC:  Workload  Reasonability. 

EXPLANATIONS:  Mission  data  must  often  be  added  to  system  soft¬ 

ware.  To  do  this  as  efficiently  as  possible,  the  operator  must  be 
given  control  over  selecting  the  most  efficient  manner  to  perform  the 
task. 

EXAMPLES:  A  strike  planning  officer  must  enter  150  coordinates  in 

a  table.  Each  coordinate  must  be  preceded  by  the  coordinate  number 
and  latitudes  and  longitudes  must  be  entered  with  East  or  West, 
North  or  South,  including  degree  signs  and  decimal  points.  This 
procedure  is  a  time  consuming  task.  He  selects  "automatic"  mode  to 
speed  the  task.  Coordinate  numbers  are  now  automatically  sequenc¬ 
ed,  all  latitudes  are  automatically  "East"  and  longitudes  are  "North." 
The  machine  enters  the  degree  signs  and  decimal  points  for  him. 

GLOSSARY: 


OM-31 


QUESTION  DATA  SHEET 


Question  Number  32 


QUESTION :  Data  preparation  is  usually  performed  using  on-line 
devices. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Preparation  of  data  using  off-line  devices  results 
in  using  some  transfer  medium  (such  as  punch  cards  or  magnetic 
tape)  to  transfer  the  prepared  data  from  the  off-line  device  to  the 
computer.  This  procedure  increases  the  workload. 

EXAMPLES: 


GLOSSARY: 

On-line:  connected  directly  to  the  computer  system. 

Data  preparation:  Any  entry  or  changing  of  numbers,  malfunc¬ 
tions,  etc.,  performed  by  the  operator. 

Special  Response  Instructions:  For  an  aircraft  mission  data 

base  this  is  not  a  desirable  feature.  Data  preparation  should  defin¬ 
itely  be  done  away  from  the  aircraft,  probably  in  the  strike  planning 
shop. 


OM-32 


QUESTION  DATA  SHEET 


Question  Number  33 


QUESTION:  The  system  will  accept  free-format  commands  and  data. 
CHARACTERISTIC:  Workload  reasonability . 

EXPLANATIONS:  Acceptance  of  free-format  commands  and  data 
decreases  operator  workload  by  decreasing  the  requirement  for  the 
operator  to  meet  rigid,  inflexible  formatting  standards. 

EXAMPLES :  A  voltage,  hours  to  destination,  or  angle  of  attack 
value  (e.g.,  five)  that  the  operator  must  enter  is  acceptable  in  any 
of  the  following  formats,  including  leading  and  trailing  blanks:  "5", 
"5."  "5.0",  "5.00",  "05",  "05.00",  "0.5  x  10**1",  ".5E+01",  "5D+00." 

GLOSSARY :  Free-format:  any  legitimate  representation  is  accept¬ 

able. 


OM-33 


QUESTION  DATA  SHEET 


Question  Number  34 


QUESTION :  Menu  techniques  are  used  to  aid  the  operator  in  making 
decisions. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Having  the  machine  list  possible  alternatives  to  the 
operator  when  an  operator  decision  is  called  for,  aids  the  operator  in 
making  a  quick  decision  and  in  giving  consideration  to  each  alterna¬ 
tive.  The  operator  would  not  forget  to  consider  seldom  used  func¬ 
tions. 

EXAMPLES:  Weapons  Selection: 


A. 

Single 

D. 

Slick 

B. 

Pairs 

E. 

HI -Drag 

C. 

Ripple 

F. 

LO-Drag 

GLOSSARY :  Menu:  a  list  of  possible  alternatives  such  as  foods  for 
dinner  or  operations  to  perform. 


OM-34 


QUESTION  DATA  SHEET 


Question  Number  35 


QUESTION :  The  system  may  be  operated  without  reference  to  man¬ 
uals  during  normal  operations. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  An  experienced  operator  will  be  slowed  down  if 
forced  to  make  constant  reference  to  manuals  while  performing  tasks. 
Machines  can  be  designed  to  include  appropriate  manual-type  informa¬ 
tion  in  the  displays  presented  to  the  operator. 

EXAMPLES: 


GLOSSARY: 


OM-35 


QUESTION  DATA  SHEET 


Question  Number  36 


QUESTION :  The  operator  need  memorize  a  comfortably  small  number 
of  commands  in  order  to  effectively  operate  the  system. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  The  larger  the  number  of  commands  an  operator 
must  memorize  in  order  to  use  the  machine,  the  more  difficult  and 
the  more  error-prone  the  process. 

EXAMPLES:  Operators  which  need  to  use  thousands  of  different 
commands  (such  as  programmers)  must  make  constant  reference  to 
manuals.  This  procedure  is  a  time  consuming  task. 

GLOSSARY: 


OM-36 


QUESTION  DATA  SHEET 


Question  Number  37 


QUESTION :  Messages  to  the  operator  are  easy  to  understand. 
CHARACTERISTIC:  Workload  Reasonability. 

EXPLANATIONS:  Any  message  which  is  not  easy  to  understand 

causes  the  operator  to  expend  extra  effort  in  order  to  gain  an 
understanding. 

EXAMPLES :  "Ratio  of  actual  speed  to  speed  of  sound"  is  much  more 
difficult  to  understand  than  "Mach  Number." 

GLOSSARY: 


-3: 


QUESTION  DATA  SHEET 


Question  Number  38 


QUESTION :  The  device  used  to  send  messages  to  the  operator 
provides  information  at  a  rate  comfortable  to  the  operator. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  A  display  device  which  sends  information  to  the 
operator  too  fast  can  cause  him  to  miss  important  information,  pos¬ 
sibly  resulting  in  task  reaccomplishment  and  frustration.  A  display 
device  which  is  too  slow  causes  the  operator  to  waste  a  lot  of  time 
and  is  also  frustrating. 

EXAMPLES: 


GLOSSARY: 


OM-38 


QUESTION  DATA  SHEET 


Question  Number  39 


QUESTION :  The  number  of  messages  presented  to  the  operator  at 
one  time  is  small. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Displaying  too  much  information  to  the  operator  at 
one  time  does  nothing  more  than  add  a  confusion  factor  and  makes 
the  operator  work  harder  to  get  exactly  the  information  desired. 

EXAMPLES:  When  a  radar  system  technician  wants  nothing  more 
than  to  know  antenna  power  output,  it  is  not  desirable  to  give  him 
the  power  output  of  every  component  of  the  system. 

GLOSSARY: 


OM-39 


QUESTION  DATA  SHEET 


Question  Number  40 


QUESTION :  The  system  software  may  be  reloaded  quickly  and 
easily. 

CHARACTERISTIC:  Workload  reasonability . 

EXPLANATIONS:  Some  computerized  systems  require  the  operator  to 
load  new  or  different  software  at  regular  or  frequent  intervals.  The 
quicker  and  easier  this  can  be  done,  the  less  demanding  and  frus¬ 
trating  it  will  be  to  the  operator.  For  those  systems  where  the 
mission  software  is  reloaded  by  someone  other  than  the  prime  mission 
operator,  such  reloading  still  has  an  impact  on  the  prime  mission 
operator  in  areas  such  as  idle  time  and  workload  backlog. 

EXAMPLES:  The  operator  is  able  to  load  the  software  from  a  disk 

by  merely  setting  some  toggle  switches  and  depressing  a  "Reboot" 
button. 

GLOSSARY :  Reloaded:  reboot,  reprogram. 


QUESTION  DATS  SHEET 

Question  Number  41 

QUESTION :  The  system  software  needs  to  be  reloaded  infrequently. 
CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Regardless  of  the  size  of  the  task  to  reload  soft¬ 
ware,  the  less  often  it  must  be  done  the  less  the  operator  workload. 

EXAMPLES: 

GLOSSARY: 


OM-41 


QUESTION  DATA  SHEET 


Question  Number  42 

QUESTION :  System  warm-up  time  is  small. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  A  machine  which  requires  large  amounts  of  warm¬ 
up  time  before  use  causes  the  operator  to  waste  time,  lowers  pro¬ 
ductivity,  and  often  frustrates  the  operator. 

EXAMPLES: 

GLOSSARY: 


OM-42 


QUESTION  DATA  SHEET 


Question  Number  43 


QUESTION :  The  operator's  manual  makes  minimal  use  of  cross-refer¬ 
ences. 

CHARACTERISTIC :  Workload  Reasonability. 

EXPLANATIONS:  Excessive  cross-referencing  in  the  operators 

manual  forces  extra  work  on  the  operator.  The  worst  case  happens 
when  the  operator  has  used  all  fingers  and  toes  and  forgetten  the 
original  question.  Cross-referencing  should  be  restricted  to  cases 
where  it  serves  a  useful  purpose. 

GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  44 


QUESTION :  It  is  easy  to  locate  specific  information  within  the  oper¬ 

ator's  manual. 

CHARACTERISTIC:  Workload  Reasonability. 

EXPLANATIONS:  Manuals  which  an  operator  must  or  might  use 

during  operation  of  a  machine  should  be  arranged,  indexed,  and 
have  a  table  of  contents  such  that  the  operator  can  locate  specific 
information  rapidly. 

EXAMPLES:  The  evaluator  should  think  up  some  information  desir¬ 

able  to  retrieve  from  the  operators  manual,  then  see  how  hard  it  is 
to  find  it.  This  process  may  be  repeated  several  times. 

GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  45 

QUESTION :  The  operator's  manual  is  a  reasonable  size. 
CHARACTERISTIC:  Workload  Reasonability. 

EXPLANATIONS:  Oversize  operator's  manuals  are  physically  difficult 
to  work  with  as  well  as  reflecting  poor  organization  and  lack  of  con¬ 
sideration  of  what  the  operator  needs  to  know. 

EXAMPLES: 

GLOSSARY: 


OM-45 


1 


QUESTION  DATA  SHEET 

Question  Number  46 


QUESTION :  The  operator  performs  no  tedious  functions  which  could 
be  handled  by  the  system. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Functions  which  could  be  performed  by  the 
machine  but  which  are  forced  upon  the  operator  represent  a  source 
of  additional  workload  for  the  operator. 

EXAMPLES:  1.  Hand  copying  of  information  from  computer  display 
to  records. 

2.  Entry  of  degree  signs  and  decimal  points. 

3.  Repeated  entry  of  duplicated  data,  such  as  the 
words  "Year"  and  "Month"  when  entering  dates. 

GLOSSARY: 


OM-46 


QUESTION  DATA  SHEET 


Question  Number  47 


QUESTION :  The  operator  is  rarely  bored  and  performs  a  dynamic 
function . 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  A  well-designed  system  takes  into  consideration 
the  best  capabilities  of  the  computer  (repetitive  tasks)  and  the  best 
capabilities  of  the  operator  (decision  making,  experience). 

EXAMPLES:  The  old  (obsolete)  "smart  machine-dumb  man"  concept 
disallowed  the  operator  to  make  decisions  which  could  have  resulted 
in  considerable  time  savings  and  increased  productivity,  such  as 
skipping  tests  which  the  operator  already  knew  the  "black  box" 
would  pass. 

GLOSSARY: 


OM-47 


QUESTION  DATA  SHEET 


Question  Number  48 


QUESTION :  The  operator  is  not  forced  to  wait  for  the  machine  to 
respond . 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  An  operator  who  is  continuously  forced  to  wait  for 
the  machine  to  become  available  for  use  is  less  than  optimally  produc¬ 
tive  and  is  probably  frustrated.  Machines  which  respond  slowly  are 
probably  overloaded.  Whenever  an  operator  is  forced  to  wait  for  a 
machine  to  respond,  valuable  human  resources  are  being  wasted. 

EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  49 


QUESTION :  The  operator  is  not  a  slave  to  the  machine. 
CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Operators  which  are  "chained"  to  a  machine  get 

worn  out  rapidly,  tend  to  become  inattentive,  and  make  more  and 
more  errors  as  time  passes.  Furthermore,  they  have  little  job  satis¬ 
faction  and  have  a  high  turn-over  rate,  resulting  in  increased  train¬ 
ing  costs  and  low  organizational  productivity. 

EXAMPLES:  Automobile  assembly-line  workers  are  perfect  examples 

of  operators  "chained"  to  a  machine. 

GLOSSARY: 


OM-49 


WORKLOAD  REASONABILITY.  A  computerized  system  contain  the 
quality  of  workload  reasonability  to  the  extent  that  the  tasks  require- 
ed  of  the  operator  are  within  the  operator's  capabilities  and  the 
extent  to  which  the  operator  performs  a  useful,  meaningful  role. 

An  Optimum  design  of  a  system  which  involves  an  operator  and 
a  computerized  machine  takes  advantage  of  the  best  capabilities  of 
both:  the  machine  to  perfor  repetitive  tasks  rapidly  and  the  oper¬ 
ator  to  make  command  decisions  involving  unusual  situations. 


D*  R IPTJ_Y,EN ESS .  A  computerized  system  contains  the  quality 

of  descriptiveness  to  the  extent  that  the  operator  has  available 
adequate  of  every  function  the  operator  is  required  to  perform,  and 
every  function  the  machine  performs. 

The  operator  need  not  be  informed  in  detail  of  every  task  the 
machine  performs,  but  there  definitely  are  certain  things  the  oper- 
*novV°  fulfil  the  mission.  The  questionnaire  relies  upon 
the  knowledge  of  the  operator  to  define  what  it  is  the  operator  needs 
to  Know. 


QUESTION  DATA  SHEET 


Question  Number  50 


QUESTION :  Power-on  and  power-off  procedures  are  well-docu¬ 
mented. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Since  the  machine  cannot  aid  the  operator  during 
power-on  and  power-off  activities,  an  easy-to-use,  understandable, 
and  correct  set  of  written  procedures  must  be  available  to  the  oper¬ 
ator. 

EXAMPLES: 


GLOSSARY: 


OM-50 


QUESTION  DATA  SHEET 


Question  Number  51 


QUESTION :  The  operator  has  adequate  instructions  for  handling 
emergencies. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  The  operator  must  have  adequate  emergency  in¬ 
structions  available  if  he  is  to  avoid  harm  to  personnel  or  damage  to 
equipment. 

EXAMPLES: 


GLOSSARY: 


OM-51 


QUESTION  DATA  SHEET 

Question  Number  52 


QUESTION :  Legitimate  responses  for  all  conditions  are  explained. 
CHARACTERISTIC :  Descriptiveness. 

EXPLANATIONS:  In  order  for  a  system  to  be  considered  properly 
described,  alj  situations  which  the  normal  operator  will  encounter 
must  be  addressed  by  either  the  documentation  or  by  computer¬ 
generated  descriptions. 

EXAMPLES: 


GLOSSARY :  Explained:  documented  and/or  prompted  by  the  soft¬ 

ware. 


OM-52 


QUESTION  DATA  SHEET 


Question  Number  53 


QUESTION :  The  software  provides  a  question-  answer-type  aid. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  The  availability  of  question-answer  operator  aids 
can  greatly  aid  operator  work  by  making  technical  information  quickly 
and  readily  available. 

EXAMPLES: 

1.  One  system  uses  a  "help  me"  command.  If  the  operator 
needs  information  on  how  to  format  a  tape  advance  command,  for 
instance,  he  enters  "HELP  ME,  TAPE  ADVANCE"  and  he  is  instantly 
presented  with  a  detailed  explanation  of  various  tape  control  com¬ 
mands. 

2.  Another  system  responds  with  a  more  detailed  data  request 
whenever  the  operator  enters  a  question  mark.  For  instance  the 
operator  leaves  the  machine  and  later  returns  to  find  the  machine 
requesting  "ENTER  ALTITUDE."  The  operator  has  forgotten  which 
altitude  was  being  requested  when  he  left.  A  question  mark  is 
entered  and  the  machine  responds  "ENTER  SIMULATED  AIRCRAFT 
ALTITUDE  IN  THOUSANDS  OF  FEET." 

GLOSSARY: 


OM-53 


QUESTION  DATA  SHEET 


Question  Number  54 


QUESTION :  The  system  will  explain  each  command  upon  user 
request. 

CHARACTERISTIC :  Descriptiveness. 

EXPLANATIONS:  Descriptions  of  operator  commands  should  be 
readily  available  to  the  operator.  This  is  especially  helpful  when  the 
operator  makes  regular  use  of  a  very  large  number  of  commands. 

EXAMPLES:  One  system  automatically  displays  a  short  explanation  of 
what  is  required  anytime  an  illegal  command  is  entered.  Therefore, 
anytime  the  operator  needs  more  information  before  making  his  deci¬ 
sion^),  he  enters  an  "X"  or  other  illegal  command.  The  machine 
then  gives  him  a  short  summary  of  legal  actions  and  automatically 
re-asks  the  original  question. 

GLOSSARY: 


OM-54 


QUESTION  DATA  SHEET 


Question  Number  55 


QUESTION :  Explanations  of  how  to  interpret  all  output  data  are 
available. 

CHARACTERISTIC:  Descriptiveness . 

EXPLANATIONS :  Data  without  explanations  is  often  useless.  There¬ 
fore,  anytime  information  is  displayed  to  the  operator,  the  operator 
should  be  able  to  request  and  receive  additional  explanatory  informa¬ 
tion  to  help  him  interpret  the  actual  meaning  of  the  data. 

EXAMPLES :  Explanations  may  be  available  through  an  on-line  ques¬ 
tion-type  aid  or  through  the  operators  manual. 

GLOSSARY: 


OM-55 


QUESTION  DATA  SHEET 


Question  Number  56 


QUESTION :  The  operator  is  adequately  alerted  when  the  system 
requires  operator  action. 

CHARACTERISTIC :  Descriptiveness. 

EXPLANATIONS:  When  the  machine  is  performing  a  long  task  which 
requires  no  operator  actions,  the  operator  may  do  something  else, 
such  as  take  a  coffee  break  or  carry  on  a  conversation.  If  no  time 
is  to  be  wasted,  however,  he  must  be  told  when  to  end  the  break. 

EXAMPLES: 

GLOSSARY: 

Alerted: 

Optimum:  a  combination  of  audible  and  visual  signals. 

Acceptable:  either  audible  or  visual  prompts. 

Unacceptable:  no  signal  other  than  apparent  machine 

inactivity. 
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QUESTION  DATA  SHEET 


Question  Number  57 


QUESTION :  The  machine  gives  the  operator  decision  aids  if  tasks 
cannot  be  executed  as  ordered. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  When  a  task  cannot  be  performed  as  ordered  by 
the  operator,  the  machine  should  display  information  to  aid  the 
operator  in  making  a  decision  about  alternative  ways  to  get  the  job 
done. 

EXAMPLES: 

1.  If  an  automatic  test  equipment  operator  has  a  test  stop 
because  a  certain  power  supply  is  not  installed  in  the  test  equip¬ 
ment,  then  the  machine  should  list  other  power  supplies  available  for 
replacements.  The  operator  would  then  be  able  to  choose  a  different 
power  supply  and  continue  the  test. 

2.  On  a  simulator,  the  machine  should  tell  the  operator  exactly 
what  procedures  to  use  to  simulate  a  partial  mission  if  one  of  the 
student  stations  is  inoperative. 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  58 


QUESTION :  The  version  number  (revision  number)  of  the  software 
is  readily  available  to  the  operator. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Frequent  software  updates  dictate  that  the  opera¬ 
tor  be  aware  of  exactly  which  version  of  an  operational  program  is 
being  used. 

EXAMPLES: 

1.  An  automatic  test  equipment  operator  must  usually  use  a 
new  test  program  after  a  TCTO  has  been  implemented  on  a  given 
black  box.  Modified  black  boxes  will  often  fail  an  old  test  program 
even  though  they  are  good. 

2.  In  simulators,  due  to  frequent  software  changes,  there 
often  is  not  a  specific  version  number  associated  with  loaded  soft¬ 
ware.  Rather,  there  may  be  a  status  board  available  to  the  instruc¬ 
tor  pilot  which  indicates  the  most  recent  software  updates  and  what 
they  mean. 

GLOSSARY: 


OM-58 
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QUESTION  DATA  SHEET 


Question  Number  59 


QUESTION :  Data  base  configuration  data  is  readily  available  to  the 
operator. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Many  systems  use  data  bases,  the  configuration  of 
which  is  as  important  as  the  configuration  of  the  computer  programs. 

EXAMPLES: 

1.  Simulators:  The  flight  characteristics  of  a  given  aircraft 
may  exist  as  a  data  base. 

2.  Automatic  Test  Equipment:  Parameters  of  stimulus  and 
measurement  devices  may  exist  as  a  data  base. 

3.  Mission  Data  Preparation:  Mission  data  tapes  of  latitudes 
and  longitudes  from  a  data  base  containing  thousands  or  millions  of 
mensurated  points. 

GLOSSARY:  Data  base:  a  collection  of  numeric  parameters  which 

describe  an  entity  such  as  flight  characteristics  of  an  aircraft  or  the 
contents  of  all  accounts  with  a  credit  union. 


OM-59 


QUESTION  DATA  SHEET 


Question  Number  60 


QUESTION :  All  documents  the  operator  requires  (including  cross- 
references)  are  easily  available  to  him. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  The  operator  will  be  unable  to  perform  his  duties 
if  he  does  not  have  available  documents  which  contain  the  infor¬ 
mation  he  needs. 


EXAMPLES: 


GLOSSARY: 


OM-60 


f 


QUESTION  DATA  SHEET 


Question  Number  61 


QUESTION :  The  operator's  manual  clearly  explains  the  normal 
sequential  steps  of  operation. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Lack  of  descriptions  of  normal  sequential  steps  of 
operation  results  in  poor  training  and  in  operator  uncertainty. 
Poorly  organized  manuals  make  the  operator  hunt  and  dig  through  all 
the  information  available  to  figure  out  what  may  be  simple,  straight¬ 
forward  "normal"  procedures.  The  normal  sequential  steps  of  opera¬ 
tions  should  not  be  hidden  amid  verbiage  describing  unusual  situa¬ 
tions. 

EXAMPLES: 


GLOSSARY: 


OM-61 


QUESTION  DATA  SHEET 


Question  Number  62 


QUESTION :  The  operator's  manual  contains  a  useful  table  of 

contents. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  A  table  of  contents  is  the  key  to  the  organization 
and  content  of  an  operator's  manual. 

EXAMPLES: 


GLOSSARY :  Useful :  The  table  of  contents  not  only  exists,  but 

actually  reflects  the  organization  and  content  of  the  operator's 
manual . 


OM-62 


QUESTION  DATA  SHEET 


Question  Number  63 


QUESTION :  The  operator's  manual  contains  a  useful  index. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  An  index  is  required  to  allow  the  operator  to 

locate  specific  information  within  the  operator's  manual  by  subject 
matter.  Important  subject  matter  often  does  not  appear  in  the  table 
of  contents  under  a  heading  the  operator  is  familiar  with. 

EXAMPLES:  The  table  of  contents  may  have  a  300  page  section 

titled  "Malfunctions,"  whereas  the  operator  would  find  "engine  bleed 
air  malfunctions"  through  the  index  without  leafing  through  the  300 
pages. 

GLOSSARY: 


OM-63 


QUESTION  DATA  SHEET 
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Question  Number  64 


QUESTION :  The  operator's  manual  contains  a  useful  glossary. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  A  glossary  of  terminology  is  often  essential  to  an 
operator's  manual  which  is  clear,  concise,  and,  above  all,  descriptive. 

EXAMPLES: 


GLOSSARY: 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


DESCRIPTIVENESS.  A  computerized  system  contains  the  quality  of 
descriptiveness  of  the  extent  that  the  operator  has  available  adequate 
explanations  of  every  function  the  operator  is  required  to  perform, 
and  every  function  the  machine  performs.  The  operator  need  not  be 
informed  in  detail  of  every  task  the  machine  performs,  but  there 
definitely  are  certain  things  the  operator  must  know  to  fulfil  the 
mission.  The  questionnaire  relies  upon  the  knowledge  of  the  oper¬ 
ator  to  define  what  it  is  the  operator  needs  to  know. 


E.  CONSISTENCY.  A  computerized  system  contains  the  quality  of 
consistency  to  the  extent  that  the  behavior  of  the  machine  and 
documentation  corresponds  to  the  expectations  of  the  operator. 

There  should  be  a  near  one-to-one  correspondence  between 
what  the  machine  does,  what  the  documentation  says  it  will  do,  and 
what  the  operator  has  been  trained  to  expect  the  machine  to  do. 
Furthermore,  the  documentation  normally  available  to  the  operator 
should  agree  with  that  which  the  operator  has  been  trained  to 
expect. 


QUESTION  DATA  SHEET 


Question  Number  65 


QUESTION :  Operator  entered  commands  are  systematically  formatted. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  To  operate  the  machine,  the  operator  must  mem¬ 
orize  a  set  of  command  instructions.  The  more  systematic  the  naming 
convention  and  format  used  for  the  commands,  the  easier  it  is  to 
memorize  and  use. 

EXAMPLES: 


GLOSSARY:  Systematically  formatted:  It  should  be  obvious  that  the 
software  designers  used  a  naming  convention  and  formatting  conven¬ 
tion  in  designing  the  set  of  commands  which  an  operator  must  enter. 


OM-65 


QUESTION  DATA  SHEET 


Question  Number  66 


QUESTION:  The  command  language  is  a  standardized  language. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  Use  of  a  standardized  language  has  several  advan¬ 
tages: 

1.  The  new  operator  may  already  know  it. 

2.  There  are  usually  better  manuals  available. 

3.  There  are  usually  more  people  around  who  can  offer 
assistance. 

EXAMPLES: 

1.  Data  processing:  COBOL 

2.  Scientific  applications:  FORTRAN 

3.  Mini-computers:  BASIC 

4.  Simulators:  English 

GLOSSARY:  Command  Language:  The  set  of  commands  which  an 

operator  may  enter.  In  programming  environments,  the  operator 
must  usually  know  at  least  two  command  languages;  the  programming 
language  and  the  operating  system  language. 


OM-66 


QUESTION  DATA  SHEET 


Question  Number  67 


QUESTION :  Requirements  for  operator  input  agree  with  the  oper¬ 

ator's  manual. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  When  there  is  inconsistency  between  the  operators 
manual  and  actual  operations,  it  leads  to  operator  confusion,  frustra¬ 
tion,  errors,  and  re-accomplishment. 

EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  68 


QUESTION :  Messages  to  the  operator  are  systematically  formatted. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  When  messages  to  the  operator  are  systematically 
formatted  they  aid  the  operator  in  that  he  can  read  faster  and  always 
knows  where  to  look  within  a  message  to  get  exactly  the  information 
desired. 

EXAMPLES: 


GLOSSARY:  Systematically  Formatted:  It  should  be  obvious  that 

the  software  designers  used  both  a  naming  convention  and  a  format¬ 
ting  convention. 
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QUESTION  DATA  SHEET 


Question  Number  69 


QUESTION :  Messages  requiring  action  by  the  operator  are  always 
highlighted  in  some  fashion. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  When  the  machine  requires  operator  action  before 
it  can  continue  processing,  it  must  inform  the  operator.  To  minimize 
the  necessity  for  the  operator  to  constantly  watch  the  machine,  the 
machine  must  tell  the  operator  every  time  it  needs  operator  attention, 
and  it  should  always  do  it  in  the  same  manner  so  that  the  operator 
knows  what  to  expect. 

EXAMPLES:  A  combination  of  audio  and  visual  signals  is  best. 

Either  audio  or  visual  is  next  best  in  any  case,  the  operator  should 
always  be  alerted  in  the  same  manner. 

GLOSSARY: 


OM-69 


QUESTION  DATA  SHEET 


Question  Number  70 


QUESTION :  Operator  entries  always  result  in  some  type  of  re¬ 
sponse. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  For  the  operator  to  know  that  the  machine  has 
deciphered  the  operator's  commands  and  is  taking  action  upon  them, 
the  machine  must  so  indicate  to  the  operator. 

EXAMPLES:  One  system  responds  with  an  asterisk,  another  simply 

moves  the  display  cursor.  Vet  another  system  replies  with 
"COMMAND"  each  time  it  has  completed  action  on  the  previous  com¬ 
mand  . 

GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  71 


QUESTION :  Response  times  are  similar  for  groups  of  similar  activi¬ 
ties. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  To  minimize  operator  frustration,  the  machine  must 
"behave"  similarly  in  similar  situations.  Response  time  is  an  impor¬ 
tant  factor  in  measuring  similar  behaviors. 

Another  system  which  would  minimize  operator  frustration  con¬ 
cerning  response  time  would  be  for  the  system  to  display  to  the 
operator  the  amount  of  time  remaining  before  the  system  would  re¬ 
quire  operator  action. 

EXAMPLES:  When  performing  any  particular  task,  it  is  better  for 
the  operator  to  know  that  the  machine  will  always  finish  the  task  in 
about  5  seconds  than  it  is  for  him  to  know  that  the  machine  may 
finish  the  task  in  anywhere  from  .5  second  to  5  seconds. 

The  latter  situation  causes  him  to  wait  expectantly  for  up  to  5 
seconds;  whereas,  in  the  former  situation,  he  already  "knows"  when 
it  will  respond. 

Remote  terminal  systems  are  particularly  bad  about  this.  Some 
systems  have  response  times  that  can  vary  between  .1  seconds  and 
20  minutes. 

GLOSSARY: 


OM-71 


QUESTION  DATA  SHEET 


Question  Number  72 


QUESTION :  System  performance  corresponds  with  documented  per¬ 
formance  (specifications,  user's  manuals,  etc.). 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  Much  instruction  and  almost  all  troubleshooting 
depends  heavily  upon  documentation.  If  machine  performance  does 
not  agree  with  what  is  documented,  then  the  value  of  formal  training 
is  decreased,  and  troubleshooting  efforts  (as  well  as  daily  operations) 
are  made  more  difficult. 

EXAMPLES: 


GLOSSARY: 
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Question  Number  73 


QUESTION :  Checklists  agree  with  the  operator's  manual. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  A  checklist  should  be  a  summary  of  essential 

information  contained  in  the  operator's  manual.  It  should  not  be 
developed  independently  of  the  operator's  manual. 

EXAMPLES: 


GLOSSARY: 
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Question  Number  74 


QUESTION :  Operator's  manuals  are  systematically  formatted. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  In  cases  where  there  is  more  than  one  operator's 

manual,  all  such  manuals  should  be  laid  out  the  same  way.  Thus 
operators  who  must  use  more  that  one  such  manual  may  pay  attention 
to  content  rather  than  wasting  tme  trying  to  locate  information. 

EXAMPLES:  There  is  a  standard  for  the  layout  of  aircraft  -1 

manuals. 


GLOSSARY: 


CONSISTENCY .  A  computerized  system  contains  the  quality  of 
consistency  to  the  extent  that  the  behavior  of  the  machine  and 
documentation  corresponds  to  the  expectations  of  the  operator. 
There  should  be  a  near  one-to-one  correspondence  between  what  the 
machine  does,  what  the  documentation  says  it  will  do,  and  what  the 
operator  has  been  trained  to  expect  the  machine  to  do.  Furthermore, 
the  documentation  normally  available  to  the  operator  should  agree 
with  that  which  the  operator  has  been  trained  to  expect  to  see. 


F.  SIMPLICITY.  A  computerized  system  contains  the  quality  of 
simplicity  to  the  extent  that  information  presented  to  the  operator  or 
entered  by  the  operator  is  grouped  into  short,  readily  understand¬ 
able  structures. 

Complicated  data  structures,  data  entry  formats,  or  operator's 
manuals  all  require  the  operator  to  spend  more  time  in  developing  an 
understanding  of  the  system,  and  may  have  a  tendency  to  confuse 
the  operator  as  well. 


QUESTION  DATA  SHEET 


Question  Number  75 


QU  ESTION :  The  operator  needs  to  know  only  one  command  lan¬ 
guage. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  The  less  the  operator  needs  to  know,  the  easier  it 
is  to  know  it  well.  If  he  must  know  two  command  languages  to 
operate  the  system,  the  task  of  language  learning  is  about  twice  as 
complex . 

EXAMPLES: 

1.  Data  processing:  The  operating  system,  test  editor, 

interactive  compiler,  and  text  listing  programs  may  use  four  different 
commands  for  what  is  basically  the  same  function. 

2.  Simulators:  The  command  language  used  at  various  instruc¬ 
tor  stations  should  be  similar  or  the  same. 

3.  Automatic  Test  Equipment:  Commands  used  to  test  various 
LRUs  should  be  the  same  where  the  function  of  the  command  is  the 
same. 

GLOSSARY: 
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Question  Number  76 

QUESTION :  Operator  entered  instructions  are  relatively  short. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  In  generally,  the  shorter  an  instruction  the  easier 
it  is  to  formulate  because  it  contains  fewer  ideas  which  must  be 
combined  to  build  the  instruction. 

EXAMPLES: 

GLOSSARY: 


OM-76 


QUESTION  DATA  SHEET 


Question  Number  77 


QUESTION :  It  is  easy  to  understand  actions  required  of  the  oper¬ 

ator. 

CHARACTERISTIC:  Simplicity: 

EXPLANATIONS:  In  order  to  make  it  easy  for  the  operator  to 

understand  what  is  supposed  to  be  accomplished,  the  instructions 
must  be  relatively  simple  and  easy  to  understand. 

EXAMPLES: 


GLOSSARY: 
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QUESTION  DATA  SHEET 


Question  Number  78 


QUESTION :  Messages  to  the  operator  are  short. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Shorter  messages  generally  contain  less  information 
than  long  messages.  The  less  information  contained  in  a  message, 
the  easier  it  is  understood. 

EXAMPLES: 


GLOSSARY: 


Oi 


QUESTION  DATA  SHEET 


Question  Number  79 

QUESTION :  Each  new  message  contains  only  one  idea  to  which  the 
operator  must  respond. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  The  simplest  message  is  one  which  contains  only 
one  idea.  The  simpler  a  message,  the  easier  it  is  to  understand. 

EXAMPLES: 

GLOSSARY: 


OM-79 


QUESTION  DATA  SHEET 


Question  Number  80 


QUESTION :  Only  essential  or  useful  information  is  displayed  to  the 
operator. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS :  Excess  information  makes  it  more  difficult  to  find 
required  information  and  the  non-essential  information  may  confuse 
the  operator.  The  amount  of  data  displayed  to  the  operator  should 
not  be  a  hinderance  to  understanding. 

EXAMPLES: 


GLOSSARY: 


OM-80 


QUESTION  DATA  SHEET 


Question  Number  81 


QUESTION :  The  display  is  not  overcrowded  (unless  commanded  to 
be  so). 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Crowded  displays  decrease  the  ability  to  organize 
the  information  displayed.  The  less  organized  a  display,  of  course, 
the  harder  it  is  to  find  useful  information  therein. 

EXAMPLES: 


GLOSSARY: 


OM-81 


QUESTION  DATA  SHEET 


Question  Number  82 


QUESTION :  Difficult  words  or  characters  are  rarely  used. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Using  a  hard-to-understand  word  or  symbol  in  a 
message  makes  the  whole  message  hard  to  understand,  multiplying 
the  basic  tasks  the  operator  must  perform  to  make  his  understanding 
complete  and  accurate. 

EXAMPLES: 


GLOSSARY: 


OM-82 


QUESTION  DATA  SHEET 


Question  Number  83 

QUESTION :  Data  structures  are  easily  understandable. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  For  an  operator  to  be  able  to  rapidly  withdraw 
necessary  data  from  a  data  structure,  the  structure  of  the  data  must 
enhance  understanding. 

EXAMPLES: 


GLOSSARY:  Data  structure:  a  list  or  table  of  data,  such  as  a  mal¬ 
function  list  or  parameter  list. 


OM-83 


QUESTION  DATA  SHEET 


Question  Number  84 


QUESTION:  The  operator  has  appropriate  checklists  available. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Excess  information  makes  understanding  of  es¬ 
sential  information  difficult.  For  an  experienced  operator,  checklists 
should  represent  essential  information  and  detailed  procedures 
manuals  represent  an  excess  of  information. 

EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number  85 


QUESTION:  The  number  of  checklists  required  is  manageable. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS :  A  single  checklist  is  the  ideal  situation  for  a  given 
operator's  station.  No  operator  should  be  required  to  use  more  than 
two  or  three  checklists  simultaneously. 

EXAMPLES: 


GLOSSARY: 


OM-85 


QUESTION  DATA  SHEET 


Question  Number  86 


QUESTION :  The  operator's  manual  is  a  single  volume  (except  for 

checklists). 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Multi-volume  operator's  manuals  represent  a  level 

of  complexity  higher  than  single-volume  operator's  manuals.  Clas¬ 
sified  second  volumes  are  necessary,  but  still  more  complex  than 
single  volumes. 

EXAMPLES: 


GLOSSARY: 


OM-86 


QUESTION  DATA  SHEET 


Question  Number  87 


QUESTION :  The  operator's  manual  is  easy  to  understand. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  In  general,  the  easier  an  operator's  manual  is  to 

understand,  the  easier  it  is  to  use,  the  more  it  will  be  used,  and 
the  more  it  will  be  used  correctly. 

EXAMPLES: 


GLOSSARY: 


OM-87 


ifciHTMi 


QUESTION  DATA  SHEET 


Question  Number  88 


QUESTION :  Alternatives  to  normal  operating  sequences  are  de¬ 
scribed  separately  (not  embedded  within  normal  procedures). 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Excess  information  makes  understanding  of  es¬ 
sential  information  difficult.  Alternative  procedures  usually  repre¬ 
sent  excess  information  if  contained  within  the  normal  procedures 
(which  hopefully  are  most  often  used).  Alternative  procedures 
should,  therefore,  be  contained  in  a  separate  section  of  the  docu¬ 
mentation  . 

EXAMPLES: 


GLOSSARY: 


OM-88 


1 


SIMPLICITY.  A  computerized  system  contains  the  quality  of  simplic¬ 
ity  to  the  extent  that  information  presented  to  the  operator  or 
entered  by  the  operator  is  grouped  into  short,  readily  understand¬ 
able  structures. 

Complicated  data  structures,  data  entry  formats  or  operators 
manuals  all  require  more  the  operator  to  spend  more  time  in  develop¬ 
ing  an  understanding  of  the  system,  and  may  have  a  tendency  to 
confuse  the  operator  as  well. 


j 
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G.  GENERAL  QUESTIONS:  The  following  section  contains  general 
questions  about  characteristics  of  the  system.  The  evaluator  should 
formulate  a  subjective  opinion  based  on  general  feelings  without 
reference  to  previous  answers. 


\ 
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QUESTION  DATA  SHEET 


Question  Number  89 


QUESTION :  The  concepts  of  Assurability  as  implemented  in  the 

system  contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  Questions. 

EXPLANATIONS: 

GLOSSARY :  Assurability:  A  computerized  system  contains  the 

quality  of  assurability  to  the  extent  that  it  aids  the  operator  in 
validating  data,  avoiding  errors,  and  correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error 
avoidance  may  or  may  not  have  good  assurability.  A  system  should 
also  be  designed  so  that  errors  are  easy  to  correct,  and  above  all  so 
that  errors  do  not  have  catastrophic  effects. 


OM-89 


QUESTION  DATA  SHEET 


Question  Number  90 


QUESTION :  The  concepts  of  Controllability  as  implemented  in  the 

system  contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  Questions. 

EXPLANATIONS: 

GLOSSARY :  Controllability:  A  computerized  system  contains  the 

quality  of  controllability  to  the  extent  that  it  allows  the  operator  to 
direct  the  operations  of  the  machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of 
the  machine  in  order  to  utilize  it  effectively  and  effeciently. 


QM-90 


QUESTION  DATA  SHEET 


Question  Number  91 


QUESTION :  The  concepts  of  Workload  Reasonability  as  implemented 

in  the  system  contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  Questions. 

EXPLANATIONS: 

GLOSSARY :  Workload  reasonability.  A  computerized  system  contains 
the  quality  of  workload  reasonability  to  the  extent  that  the  tasks 
required  of  the  operator  are  within  the  operator's  capabilities  and 
the  extent  ot  which  the  operator  performs  a  useful,  meaningful  role. 

Optimum  design  of  a  system  which  involves  an  operator  and  a 
computerized  machine  takes  advantage  of  the  best  capabilities  of 
both:  the  machine  to  perform  repetitive  tasks  rapidly  and  the  oper¬ 
ator  to  make  command  decisions  involving  unusual  situations. 


QUESTION  DATA  SHEET 


Question  Number  92 

QUESTION :  The  concepts  of  Descriptiveness  as  implemented  in  the 

system  contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Descriptiveness:  A  computerized  system  contains  the 

quality  of  descriptiveness  to  the  extent  that  the  operator  has  avail¬ 
able  adequate  explanations  of  every  function  the  operator  is  required 
to  perform,  and  every  function  the  machine  performs. 

The  operator  need  not  be  informed  in  detail  of  every  task  the 
machine  performs,  but  there  definitely  are  certain  things  the  oper¬ 
ator  must  know  to  fulfil  the  mission.  The  questionnaire  relies  upon 
the  knowledge  of  the  operator  to  define  what  it  is  the  operator  needs 
to  know. 


OM-92 


QUESTION  DATA  SHEET 


Question  Number  93 


QUESTION:  The  concepts  of  Consistency  as  implemented  in  the 

system  contribute  to  the  usability  of  the  system. 

CHARACTERISTIC :  General  questions. 

EXPLANATIONS: 

GLOSSARY:  Consistency:  A  computerized  system  contains  the 
quality  of  consistency  to  the  extent  that  the  behavior  of  the  machine 
and  documentation  corresponds  to  the  expectations  of  the  operator. 

There  should  be  a  near  one-to-one  correspondence  between 
what  the  machine  does,  what  the  documentation  says  it  will  do,  and 
what  the  operator  has  been  trained  to  expect  the  machine  to  do. 
Furthermore,  the  documentation  normally  available  to  the  operator 
should  agree. 


OM-93 


QUESTION  DATA  SHEET 


Question  Number  94 


QUESTION:  The  concepts  of  Simplicity  as  implemented  in  the  system 
contribute  to  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Simplicity:  A  computerized  system  contains  the  quality 
of  simplicity  to  the  extent  that  information  presented  to  the  operator 
or  entered  by  the  operator  is  grouped  into  short,  readily  under¬ 
standable  structures. 

Complicated  data  structures,  data  entry  formats,  or  operator's 
manuals  all  require  the  operator  to  spend  more  time  in  developing  an 
understanding  of  the  system,  and  may  have  a  tendency  to  confuse 
the  operator  as  well. 


OM-94 


QUESTION  DATA  SHEET 


Question  Number  95 


QUESTION :  Overall  it  appears  that  the  operator-machine  interface 
has  been  well-designed. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS:  This  is  the  bottom-line,  most  highly  subjective 

question  asked.  Rate  the  system  according  to  your  personal  feeling. 

EXAMPLES: 


GLOSSARY: 
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SECTION  III 


REFERENCE  MATERIAL 

III. A,  Subject  Matter  Cross-Reference  (Index) 

Abort  12,18 
Aid 

Decision  34,54 
Question-Answer  53 
Alert  15,69 
Alternate  30,88 
Arrow  10 

Assurability  1-16,89 
Automatic  17,27 
Available  48,55,58-60 
Back  Arrow  10 
Backspace  10 
Bored  47 
Bypass  30,88 
Character  (Symbol)  10,82 
Checklist  73,84-86 
Comfortable  36,38 
Command 
Explain  54 
Format  33,65 
Formulation  26,76 
Language  66,75 
Memorize  36 
Modes  28 
Strings  26 

Configuration  23,58,59 
Control  24,29 
Controllability  17-30,90 
Consistency  65-74,93 
Copy  11 
Create  26 

Cross-References  43,60 
Crowded  81 
Cursor  9 
Data 

Base  14,25,59 
Change  25 
Entry  1-10,31 
Explanation  55,83 
Format  (Free-Format)  33 
Mission  31 
Preparation  32 
Recording  11 
Structures  83 
Task  31 

Verification  7,8 
Decision  Aids  34 
Descriptiveness  50-64,92 
Deselection  13 


Device 
Messages  38 
On-Line  32 
Output  20 

Selection/Deselection  13 
Difficult  82 
Display 

Cause  of  Errors  3 
Cause  of  Halt  16 
Correction  of  Error  4 
Information  80 
Status  23 

Type  of  Information  23 7 81 
Documented  50,60,72 
Dynamic  47 
Easy  5,31,37,40,44,77 
Edit  25 

Emergencies  51 
Error 
Cause  3 
Detection  2 
Correction  4,10 
Input  1-6 
Internal  14 
Explanation 

Amount  of  Text  24 
Command  54 
Legitimate  Response  52 
Of  Operation  50-55,61 
O  *put  (Data)  55 
Execution 

Actions  Prior  to  7,25 
Cannot  be  Performed  57 
Cease  14 
Stepped  27 
Failures 
Partial  30 
System  1,15,16 
Faults  15 

Format  33,65,68,74 
Free- Format  33 
Frequently  41 
Glossary  64 
Group  71 
Halt  16 

Highlighted  69 
Index  63 
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Information  Selection  22 
Locate  44 
Rate  38 
Selection  22 
Useful  80 
Infrequently  41 
Input 
Data  7,8 
Errors  1-6 
Explanations  24 
Response  to  70 
Internal  Errors  14 
Interpret  55 
Interrupt  12,17 
Instructions  51,76 
Language  65,75 

Manuals  35,42-45,50,51,60-64,67,72-74,84-88 
Menu  Techniques  34 
Memorize  36 

Messages  37-39,68,69,78,79 
Mission  Data  31 
Modes  28 
Number 

Amount  24,36,39 
Version  (Revision)  58,59 
Off-Line  13 
One  26,75,79 
Operation 
Effective  36 
Normal  35,61,88 
Operator 

Action  Required  4,56,69,77 
Aid  34,53,57 
Alert  15,56 
Commands  65,75,76 
Functions  46-49 
Input  1-6,67,70 
Messages  37-39,68,69,78 
Must  Respond  69,79 

Operators  Manual  35,43-45,61-64,67,72-74,84-88 

Out  of  Operation  13,50 

Output 

Device  Selection  20 
Explanation  of  24 
Messages  37-39,68 
Quantity  29 
Status  21,23 
Type  29 
Overcrowded  81 
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Partial  Failure  30 
Performance  72 
Pointer  8 
Power-Off  (On)  50 
Preparation  32 
Procedures  50 
Program  (Software)  40,41 
Quantity  29,36 
Question-Answer  53 
Quick  6,40,42,48 
Rarely  47 
Rate  (Speed)  38 
Recording  Data  11 
Reference  35,43,60 
Reload  Software  40 
Request  54 
Response 
Always  70 
Legitimate  52 
Time  48,71 
Uniformity  68-71 
Revision  Number  58,59 
Rubout  10 
Select 

Device  Off-Line  13 
Mode  of  Operation  28 
Self-Type  19 

Type  of  Information  22,23 
Self-Test  19 
Separate  88 
Sequential  61 
Short  76,78 
Side  Effects  12,13 
Similar  71 
Simplicity  75-88,94 
Size  45 
Slave  49 
Small  36,39,43 
Software  40,41,53,58 
Specifications  72 
Standardized  Language  66 
Status  21,23 
Strings  26 

Symbol  (Character)  10,82 
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System 

Configuration  23 
Failure  1,30 
Halt  16 

Needs  Operator  56 
Operation  35,36 
Performance  72 
Provides  Explanations  54 
Self-Test  19 
Software  40,41,58 
Status  23 
Warm-Up  42 
Systematic  65,68,74 
Table  of  Contents  62 
Task 

Abort  12 

Cannot  Be  Done  57 
Data  31 
Interrupt  12 
Tedious  46 
Time  42,71 
Understand  37,77,83 
Validity  7,8 
Verification  7,8 
Version  Number  58,59 
Visibility  27 
Wait  48 
Warm-Up  42 
Word  82 

Workload  Reasonability  31-49,91 
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SOFTWARE  OPERATOR  -  MACHINE 
INTERACE  QUESTIONNAIRE 


The  following  pages  contain  questions  about  the  operator-machine 
interface  as  it  is  affected  by  software  (computer  programs).  Each 
page  is  headed  by  the  name  of  the  characteristic  which  the  questions 
on  that  page  address.  The  definition  of  that  characteristic  follows 
the  name  of  the  characteristic.  This  definition  should  be  firmly  in 
mind  as  the  questions  on  that  page  are  answered.  Following  the 
definition  are  the  questions.  Each  question  takes  the  form  of  a 
positive  statement  about  the  operator-machine  interface  of  the  piece 
of  equipment  you  are  evaluating.  Use  the  following  scale  to  answer 
each  question  as  to  your  agreement  with  the  statement. 

A.  COMPLETELY  AGREE  (absolutely  no  doubt). 

B.  STRONGLY  AGREE. 

C.  GENERALLY  AGREE. 

D.  GENERALLY  DISAGREE. 

E.  STRONGLY  DISAGREE. 

F.  COMPLETELY  DISAGREE  (absolutely  no  doubt). 

In  addition,  one  or  more  of  the  following  standardized  comment 
responses  can  be  selected: 

I.  I  had  difficulty  answering  this  question. 

J.  A  written  comment  is  recorded  on  AFTEC  Form  207,  Com¬ 
puter  Program  Observation  Report. 

Please  answer  the  questions  carefully.  Take  as  much  time  as  you 
need.  If  there  are  any  questions  you  do  not  understand,  refer  to 
the  evaluator  guidelines,  a  copy  of  which  you  should  have  before 
you . 


SOMIQ-1 


ASSURABIUTY  QUESTIONS 


Assurability:  The  extent  that  the  system  aids  the  operator  in  vali¬ 
dating  data,  avoiding  errors,  and  correcting  errors  once  made. 

1.  Operator  input  errors  do  not  cause  system  failures. 

2.  Operator  input  errors  are  detected. 

3.  The  causes  of  input  errors  are  displayed  to  the  operator. 

4.  The  action  required  to  correct  an  operator  input  error  is  dis¬ 
played  to  the  operator. 

5.  Input  errors  are  easily  corrected. 

6.  Input  errors  are  quickly  corrected. 

7.  The  operator  can  verify  input  before  execution/entry. 

8.  Mission  peculiar  data  entered  by  the  operator  is  checked  for 

validity. 

9.  The  data  entry  display  has  a  cursor  or  pointer. 

10.  The  operator  is  able  to  correct  mistyped  characters  wi  h  a 

backspace  key. 

11.  The  system  does  not  require  the  operator  to  copy  information 
by  hand. 


12.  Task  aborts 
effects . 

and 

interrupts 

do 

not 

have 

detrimental 

side 

13. 

Selecting  a 

device 

off-line 

does 

not 

have 

detrimental 

side 

effects . 

14.  The  system  automatically  ceases  execution  if  internal  errors  are 
detected  (data-base  protection). 

15.  The  operator  is  alerted  to  faults  within  the  system. 

16.  The  causes  of  system  halts  are  displayed  to  the  operator. 


SOMIQ-2 


CONTROLLABILITY  QUESTIONS 


Controllability:  the  extent  that  the  system  allows  the  operator  to 

direct  the  operations  of  the  machine. 

17.  The  operator  can  interrupt  and  resume  automatic  processes. 

18.  The  operator  has  task  abort  capabilities  available. 

19.  The  operator  may  initiate  selected  self-test  systems. 

20.  The  operator  may  send  output  data  to  various  devices. 

21.  The  operator  can  ask  for  and  receive  the  current  status  of 

operation. 

22.  The  operator  can  select  the  type  of  information  shown  on  the 
display. 

23.  The  operator  may  command  various  displays  of  system  status. 

24.  The  operator  may  control  the  amount  of  required  explanatory 

text,  both  input  and  output. 

25.  The  operator  may  edit  the  data  base  prior  to  use  by  the  system. 

26.  The  operator  may  create  and  execute  strings  of  commands  as  a 
single  command. 

27.  The  operator  can  command  visibility  (stepped  execution)  of 

automatic  processes. 

28.  The  operator  may  command  different  modes  of  operation. 

29.  The  operator  may  control  the  type  and  quantity  of  output. 

30.  Bypass  procedures  are  available  so  that  in  cases  of  partial 
system  failure  the  more  important  system  functions  can  still  be  per¬ 
formed  . 


SOMIQ-3 


WORKLOAD  REASONABILITY  QUESTIONS 


Workload  Reasonability:  The  extent  that  the  tasks  required  of  the 
operator  are  within  the  operator's  capabilities  and  the  extent  to 
which  the  operator  performs  a  useful,  meaningful  role. 

31.  It  is  easy  to  enter  mission  (task)  peculiar  data. 

32.  Data  preparation  is  usually  performed  using  on-line  devices. 

33.  The  system  will  accept  free-format  commands  and  data. 

34.  Menu  techniques  are  used  to  aid  the  operator  in  making  deci¬ 
sions. 

35.  The  system  may  be  operated  without  reference  to  manuals 
during  normal  operations. 

36.  The  operator  need  memorize  a  comfortably  small  number  of 
commands  in  order  to  effectively  operate  the  system. 

37.  Messages  to  the  operator  are  easy  to  understand. 

38.  The  device  used  to  send  messages  to  the  operator  provides 
information  at  a  rate  comfortable  to  the  operator. 

39.  The  number  of  messages  presented  to  the  operator  at  one  time 
is  small. 

40.  The  system  software  may  be  reloaded  quickly  and  easily. 

41.  The  system  software  needs  to  be  reloaded  infrequently. 

42.  System  warm-up  time  is  small. 

43.  The  operator's  manual  makes  minimal  use  of  cross-references. 

44.  It  is  easy  to  locate  specific  information  within  the  operator's 
manual . 

45.  The  operator's  manual  is  a  reasonable  size. 

46.  The  operator  performs  no  tedious  functions  which  could  be 
handled  by  the  system. 

47.  The  operator  is  rarely  bored  and  performs  a  "dynamic"  function. 

48.  The  operator  is  not  forced  to  wait  for  the  machine  to  respond. 

49.  The  operator  is  not  a  slave  to  the  machine. 


SOMIQ-4 


DESCRIPTIVENESS  QUESTIONS 


Descriptiveness:  The  extent  to  which  the  operator  has  available 

detailed  explanations  of  every  function  the  operator  performs  and 
every  function  the  machine  performs. 

50.  Power-on  and  power-off  procedures  are  well  documented. 

51.  The  operator  has  adequate  instructions  for  handling  emer¬ 
gencies. 

52.  Legitimate  responses  for  all  conditions  are  explained. 

53.  The  software  provides  a  question-answer  type  operator  aid. 

54.  The  system  will  explain  each  command  upon  user  request. 

55.  Explanations  of  how  to  interpret  all  output  data  are  available. 

56.  The  operator  is  adequately  alerted  when  the  system  requires 
operator  action. 

57.  The  machine  gives  the  operator  decision  aids  if  tasks  cannot  be 
executed  as  ordered. 

58.  The  version  number  (revision  number)  of  the  software  is  readily 
available  to  the  operator  from  the  system. 

59.  Data  base  configuration  data  is  readily  available  to  the  operator. 

60.  All  documents  the  operator  requires  (including  cross-refer¬ 
ences)  are  easily  available  to  him. 

61.  The  operator's  manual  clearly  explains  the  normal  sequential 
steps  of  operation. 

62.  The  operator's  manual  contains  a  useful  table  of  contents. 

63.  The  operator's  manual  contains  a  useful  index. 

64.  The  operator's  manual  contains  a  useful  glossary. 
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Consistency:  The  extent  that  the  behavior  of  the  machine  and 

documentation  corresponds  to  the  expectations  of  the  operator. 

65.  Operator  entered  commands  are  systematically  formatted. 

66.  The  command  language  is  a  standardized  language. 

67.  Requirements  for  operator  input  agree  with  the  operators 
manual . 

68.  Messages  to  the  operator  are  systematically  formatted. 

69.  Messages  requiring  action  by  the  operator  are  always  high¬ 
lighted  in  some  fashion. 

70.  Operator  entries  always  result  in  some  type  of  response. 

71.  Response  times  are  similar  for  groups  of  similar  activities. 

72.  System  performance  corresponds  with  documented  performance 
(specifications,  users'  manuals,  etc.). 

73.  Checklists  agree  with  the  operator's  manual. 

74.  Operator's  manuals  are  systematically  formatted. 


SOMIQ-6 


SIMPLICITY  QUESTIONS 


Simplicity:  The  extent  that  information  presented  to  the  operator  or 
entered  by  the  operator  is  grouped  into  short,  readily  understand¬ 
able  structures. 

75.  The  operator  needs  tc  know  only  one  command  language. 

76.  Operator  entered  instructions  are  relatively  short. 

77.  It  is  easy  to  understand  actions  required  of  the  operator. 

78.  Messages  to  the  operator  are  short. 

79.  Each  new  message  contains  only  one  idea  to  which  the  operator 
must  respond. 

80.  Only  essential  or  useful  information  is  displayed  to  the  operator. 

81.  The  display  is  not  overcrowded  (unless  commanded  to  be  so). 

82.  Difficult  words  or  characters  are  rarely  used. 

83.  Data  structures  are  easily  understandable. 

84.  The  operator  has  appropriate  checklists  available. 

85.  The  number  of  checklists  required  is  manageable. 

86.  The  operator's  manual  is,  a  single  volume  (except  for  checklists). 

87.  The  operator's  manual  is  easy  to  understand. 

88.  Alternatives  to  normal  operating  sequences  are  described  sepa¬ 
rately  (not  embedded  withtn  normal  procedures). 
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GENERAL  QUESTIONS 


Note:  The  following  questions  relate  to  the  evaluator's  general 

impression  of  the  computer  program's  contribution  to 
system  usability  or  effectiveness.  Definitions  of  the  test 
factors  should  be  reviewed  before  completing  these  ques¬ 
tions. 

89.  The  concepts  of  Assurability  as  implemented  in  the  system 

contribute  to  usability  of  the  system. 

90.  The  concepts  of  Controllability  as  implemented  in  the  system 

contribute  to  usability  of  the  system. 

91.  The  concepts  of  Workload  Reasonability  as  implemented  in  the 

system  contribute  to  usability  of  the  system. 

92.  The  concepts  of  Descriptiveness  as  implemented  in  the  system 
contribute  to  usability  of  the  system. 

93.  The  concepts  of  Consistency  as  implemented  in  the  system 

contribute  to  usability  of  the  system. 

94.  The  concepts  of  Simplicity  as  implemented  in  the  system  contrib¬ 
ute  to  usability  of  the  system. 

95.  Overall  it  appears  that  the  operator-machine  interface  has  been 
well  designed. 
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C.  Example  Answer  Sheet 


A  copyrighted  form.  Reprinted  by  Permission  of  National  Computer  Sys 
terns,  Inc.,  Minneapolis,  Minnesota. 
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COMPUTER  PROGRAM  OBSERVATION  REPORT 


computer  program  identification 
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